Compare commits

..

43 Commits

Author SHA1 Message Date
c85aaf2ecb Docs: put named row access where someone leaving IfxPy will find it
The API reference gained the Row section with the release, which is the
right home for the details and the wrong place to discover the feature
exists. Three gaps closed.

The IfxPy migration guide is the page for people in exactly the position
that prompted this: IfxPy is positional-only, so those codebases grow a
description/zip helper that yields a dict and still no attribute access.
That page now shows the helper it replaces alongside the one-liner, and
notes the .lower() in the hand-written version is already a no-op since
Informix folds unquoted identifiers.

Quickstart gains one sentence, next to the paramstyle note where someone
is already thinking about how rows come back.

The phase log records 2026.09.03, including where the testing actually
found the shadowed-names bug: not in the differential test across twenty
awkward types, which found nothing, but in six lines of name bookkeeping
that were correct when written and stopped being correct an hour later.

Verified live by content, and the #rows anchor the new cross-links point
at resolves.
2026-09-03 16:06:34 -06:00
acc21b8b81 Docs: row_factory reference, and the write-path corruption warning
The API page documents Row: the three access styles, that it is set on
the connection, that it subclasses tuple so existing code is unaffected,
the measured 9% and why that makes it opt-in, and the four behaviours
worth knowing (lower-case folding, expression column names, duplicate
names, and a column beating a method of the same name).

The types page gains the placeholder-rewriter entry. It is the only one
in that list affecting the write path rather than the read path, so
anyone who has stored HH:MM times or URLs with ports through this driver
should check what actually landed.

Test counts updated to 472/471. Verified live by content.
2026-09-03 16:00:53 -06:00
9fbb97199f Release 2026.09.03 — placeholder rewriter, and named row access
The rewriter that converts :N placeholders to ? was a bare regular
expression, so it rewrote the inside of string literals. Any HH:MM time,
any URL with a port, any key:value string was silently stored wrong. It
was not opt-in either: the rewrite runs whenever a statement has
parameters, whatever placeholder style the caller used. Replaced with a
scanner that substitutes only outside quotes and comments, using
Informix's own lexical rules measured against all three servers.

Row adds column-name and attribute access alongside the existing
positional access, on request from a user coming from pyodbc and
mssql-python. Opt-in at connect time, because it costs about 9% on bulk
fetch and that is not a trade to make for someone who never reads a
column by name.

472 tests on 15 and 14.10, 471 on 12.10.
2026-09-03 15:59:26 -06:00
ce5a582048 Testing Row found the same mistake I keep making
I hand-listed the names a column could shadow: count and index, the two
tuple methods. Then I wrote keys(), _asdict() and _fields on the same
class and did not go back. A column called "keys" returned a bound
method instead of a value, silently, which is precisely the failure
shape this driver spent a fortnight removing from its decoders.

The fix is not a longer list. The reserved set is now computed from
dir(Row), so adding a method later cannot reopen the hole, and the test
is parametrized over that computed set so it grows with the class.

_fields and _map moved to name-mangled attributes. They are read by
repr() and _asdict(), so shadowing the public _fields with a column
would have made the machinery report the column value instead of the
column names. Mangling keeps the two apart, and there is a test that
shadows _fields and checks repr still works.

Also exercised, and all clean: value-identical output against the plain
tuple path across twenty columns covering every awkward type plus a
fully NULL row, which is the strongest available statement that Row is a
presentation layer and not a bug; zero-column and 500-column rows;
unicode, spaced, digit-leading, empty-string and dunder column names;
rows outliving eviction of their class from the bounded cache; the four
async fetch routes and the async pool; and twelve threads racing on the
class cache, which correctly share one class per shape.

472 tests on 15 and 14.10, 471 on 12.10.
2026-09-03 15:51:46 -06:00
805fa58eb2 Rows can answer to a column name, if you ask for it
Field request from a user running this alongside SQL Server, where both
pyodbc and mssql-python hand back rows that take a position, a column
name, and an attribute. The argument is readability on wide
projections: row[11] tells a reader nothing, and stays correct only
until somebody adds a column in the middle.

row_factory=Row gives all three. It is set on the connection, so it is
one line for the whole application rather than per query, which is what
"without any additional coding" has to mean in practice. A cursor can
override it.

Opt-in, not default, and the number is why. Measured on a 20,000-row
five-column fetch: 37.2 ms with tuples, 40.8 ms with Row, about 9%.
Defaulting it on would move the published 1.05-1.15x ratio against IfxPy
to roughly 1.15-1.25x. That is not a trade to make on behalf of somebody
running a bulk export that never looks at a column by name.

Row subclasses tuple, so row == (1, "x") still holds. That constraint
shaped the design more than anything else: this suite compares fetched
rows against plain tuples in hundreds of places, and so does everybody's
code. Slices degrade to plain tuples, since a slice has no column map.

Details that had to be decided rather than assumed, all measured against
the servers: Informix folds unquoted identifiers to lower case, so the
.lower() in the workaround people write by hand is a no-op. Expression
columns get names like "(count(*))" that cannot be attributes, so they
are subscript-only. Duplicate names resolve to the first occurrence,
matching pyodbc. And tuple already defines count and index, so a column
with either name gets a descriptor and wins, because on a database row
that is plainly what the caller meant.

The per-shape class is cached, so a type() call does not land on every
small query, and rows pickle by rebuilding from their field names.
2026-09-03 15:42:56 -06:00
088171325d The placeholder rewriter could not see a string literal
We advertise paramstyle="numeric" and the wire protocol takes ?, so :N
placeholders are rewritten on the way out. That was a bare
re.sub(r":(\d+)", "?", sql), which rewrote the inside of string literals:

    UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
                      stored as 'http://host?/x'

Any HH:MM time, any URL with a port, any aspect ratio, any key:value
string. It wrote wrong data and said nothing about it.

None of it was opt-in. The rewrite fires whenever a statement has
parameters, whatever placeholder style the caller actually used, so
writing ? everywhere and never touching numeric style did not protect
you. It also changed the placeholder count while num_qmarks was still
computed from len(params), leaving driver and server disagreeing about
how many binds exist.

Replaced with a single pass that substitutes only outside quotes and
comments. The lexical rules are Informix's own, measured against all
three servers rather than assumed from standard SQL, and one of them
would have been got wrong by habit: a backslash escapes nothing, so
'a\'b' is an unterminated string and draws -282. A scanner written to
Postgres reflexes would desync there and corrupt everything after it.
Block comments do not nest, the first */ closes them. Braces are a
comment. :: is stepped over as a unit so a cast can never be read as the
start of a placeholder.

An unterminated quote or comment consumes the rest and substitutes
nothing further. Under-substituting leaves the server to reject SQL that
was already malformed; guessing would corrupt a literal.

This was the last place the driver inferred meaning from SQL text
instead of handling it properly. Eleven of the new tests fail against
the previous commit.
2026-09-03 12:45:58 -06:00
7ffc148112 Docs: cover the 2026.09.02 review, and drop every em-dash
Content. The types page now warns about the two framing bugs this
release fixed, since both silently corrupt data on versions before it:
a BLOB or CLOB in any position but last shifted every column after it,
and a NULL collection did the same. The API page documents two things
that are now user-visible: only one scrollable cursor may be open per
connection (previously the second statement drew -285 and destroyed the
cursor as collateral), and transaction control written as SQL is tracked
(previously rollback() after a SQL BEGIN WORK sent nothing and reported
success). The phase log gains a section on the review itself, including
the pattern behind almost all eleven bugs. Test counts updated to
457/456.

Style. 136 em-dashes across 22 content files, plus Hero.astro and two
stylesheets, rewritten rather than substituted -- swapping the character
for a comma leaves prose that reads like it lost an argument with a
linter. Bullets that used a dash to gloss a term now use a colon;
parenthetical asides became their own sentences or moved inside
brackets. The rendered HTML is clean.

Verified against the live site by content, not status code: these pages
return 200 for every path and render the 404 body, so a stale deploy
looks perfectly healthy.
2026-09-02 10:36:44 -06:00
8f341c4b2a Release 2026.09.02 — a systematic review, and eleven bugs it found
No new features. The result of going back over the driver looking for
the shape of past bugs rather than for new symptoms.

Two returned wrong data silently: smart LOBs were read 77 bytes short,
shifting every column after them, and NULL composite UDTs skipped their
length field. One did nothing at all: rollback() after a SQL BEGIN WORK
returned successfully having sent nothing, leaving the rows it was asked
to discard in place and the connection back in the pool holding an open
transaction.

The rest: five ordinary query forms failing with -260 because
classification guessed from the first word; a scrollable cursor taking
the whole connection down with it; two unguarded exits leaking failed
statements; deferred cleanup landing inside somebody else's statement,
and a GC-time finalizer that couldn't tell it was already on the thread
holding the wire lock; the async layer borrowing the process-wide thread
pool; and two readers sharing one socket stream without agreeing.

457 integration tests on 15 and 14.10, 456 on 12.10 (one skip, no CTEs
before 14.10) -- up from 414. Every fix has at least one test that fails
against 2026.09.01.
2026-09-02 10:28:27 -06:00
429a3910e4 The async layer borrowed a thread pool belonging to the whole process
Every blocking call in informix_db.aio went through asyncio.to_thread,
which runs on the event loop's default executor. That executor belongs
to the process, not to us, and it is sized from the CPU count --
min(32, cpu_count + 4), so six threads on a two-CPU container.

Cancelled calls hold their threads. asyncio.to_thread cannot interrupt a
worker, so a cancelled await leaves the thread running the wire call
until the read timeout. Cancellation is ordinary in a web app -- a
client disconnect cancels the request task -- so a handful of them pins
every thread in the shared pool. Measured: six cancelled calls against a
six-worker default executor starve an unrelated to_thread indefinitely,
and with the executor held, four concurrent driver queries never ran at
all.

Pool concurrency was capped by the same number without saying so. A pool
with max_size=20 on a two-CPU box ran six queries at a time.

Each connection now owns one thread. That is the right size rather than
a compromise -- the sync connection serializes every wire operation on
its own lock, so a second thread could do nothing but wait for the
first. It also avoids a deadlock a shared pool-sized executor invites:
with N threads and N connections, N tasks blocked in acquire occupy
every thread while the connection they are waiting for is held by a task
that now needs a thread to finish and release it.

The executor lives on the sync connection so it survives being returned
to the pool and handed out again, rather than being rebuilt per acquire.
release() runs on the connection's own thread, which is idle by
definition and keeps release off any pool that waiters may have filled
-- release has to win that race, since it frees what they are waiting
for. connect() and pool acquire stay on the default executor: there is
no connection yet to own a thread, and neither can deadlock against
query threads any more.

close() shuts the executor down, and a weakref finalizer is the backstop
for a connection dropped without it -- ThreadPoolExecutor workers park
on the work queue rather than exiting when idle, so an executor that is
never shut down leaks its thread for the life of the process. The
thread-count test caught that gap; it was missing from the first cut.
2026-09-02 01:03:12 -06:00
5ad296419c Two readers shared a stream without agreeing, and a length was taken on trust
IfxSocket owns a read-ahead buffer that BufferedSocketReader fills and
drains. Connection._drain_to_eot, _raise_sq_err and the login path
bypass that reader and call IfxSocket.read_exact directly, which recv'd
from the socket without looking at the buffer. Bytes sitting in the
buffer were skipped, and skipped bytes in a length-framed protocol do
not announce themselves -- the next read lands mid-field and every read
after it is wrong.

Nothing triggers it today. The server sends one response per request, so
recv returns exactly that response and the buffered reader consumes all
of it before control returns to a direct read. That is a property of the
traffic, not of the code, and the buffer is connection-scoped precisely
so read-ahead can cross response boundaries -- pipelined executemany
already puts several responses in flight. read_exact now drains the
buffer first, which costs one branch on a cold path and makes the two
paths agree by construction rather than by luck.

fill_recv_buf believed whatever byte count it was handed, and that count
is almost always a length field straight off the wire. A garbage
0x7FFFFFFF reads as a 2 GB request and the fill loop sits in recv until
the read timeout while the buffer grows. It now refuses above
IFX_MAX_READ_BYTES (256 MiB default) with an error naming the number,
which is the actual diagnostic: a length that absurd means framing was
already lost upstream.

BufferedSocketReader.skip advanced the cursor arithmetically with no
guard, so a negative count rewound it and re-decoded consumed bytes as
the next field. The base reader's skip delegates to read_exact and does
guard; this one diverged.

Transaction control run as SQL desynced Connection._in_transaction,
which is what commit() and rollback() are guarded by and what the pool
reads to decide whether a returned connection needs cleaning up. With
autocommit on, cursor.execute("BEGIN WORK") opened a real transaction
while the flag stayed False, so rollback() returned successfully having
sent nothing and the rows it was asked to discard survived. The
connection then went back to the pool holding an open transaction and
its locks. With autocommit off it failed instead: the driver's implicit
SQ_BEGIN fired first and the caller's BEGIN WORK got -535.

The server labels these -- 34 BEGIN, 35 COMMIT, 36 ROLLBACK, with and
without the WORK keyword, measured on all three versions. JDBC reads the
same values off the describe and calls setTxBeginState/setTxEndState.
_ensure_transaction moves to after the describe, matching JDBC's
initiateTransaction placement, which is what makes it possible to skip
for transaction control at all -- until the describe lands you cannot
know that is what the statement is.
2026-09-02 00:54:12 -06:00
c67bbc4766 Statement classification guessed from the first word and got five forms wrong
The driver chose between "open a cursor and fetch" and "execute and
release" by checking whether the first word of the SQL was SELECT. That
misses a leading comment in any of the three Informix flavours (--,
/* */, { }), a parenthesized select, a parenthesized UNION, and a CTE.
All five are perfectly ordinary queries. All five failed with -260,
"Cursor name already in use" -- an error that describes neither the
cause nor anything the caller did. It says "cursor" because the driver
sent SQ_EXECUTE where the server was waiting to open one.

The server had been answering this question all along. statement_type is
the first field of the DESCRIBE response and parse_describe has always
put it in the metadata dict, where nothing read it.

The comment defending the heuristic said nfields couldn't distinguish
these cases because INSERT INTO t VALUES (?) also describes a column.
That was true, and it argued for the wrong conclusion: statement_type
distinguishes them exactly. Measured against all three servers -- every
SELECT form above reports 2, that INSERT reports 6, UPDATE 33, DELETE
32, CREATE 45, EXECUTE PROCEDURE 56.

The predicate is now JDBC's IfxSqli.isResultSet: type 2, or type 56 with
at least one column, since a routine may or may not return rows. That
last clause fixes EXECUTE FUNCTION as a side effect -- it used to run
down the DML path and discard the return value. It now yields it.

executemany keeps the first-word check as a cheap pre-flight reject, and
gains an authoritative one after PREPARE for the forms the first word
misses. The refusal releases the statement, so the connection survives.

Informix 12.10 has no CTEs and rejects WITH with -201 at offset 1. That
is the server declining the grammar rather than the driver routing it
wrongly, and the test skips on it.
2026-09-02 00:38:31 -06:00
2eb5ac8f0f The finalizer's lock probe couldn't see its own thread
The cursor finalizer runs at GC time on whatever thread happened to
allocate, so it must not touch the wire while a statement owns it. It
checked with _wire_lock.acquire(blocking=False), meaning to ask "is
anyone using the wire?" -- but an RLock grants a reentrant acquire to
its own owner. When GC fired on the thread that was mid-statement, the
probe returned True, the finalizer concluded it had exclusive access,
and it sent CLOSE/RELEASE into the middle of the statement it had just
interrupted. The victim got -208 on valid SQL.

Refcounting hid it. A dropped cursor is freed at the drop, before the
next statement runs. A cursor caught in a reference cycle waits for a
collection instead, and cycles are ordinary -- any traceback that holds
a cursor makes one. Reproduced by putting an abandoned scrollable cursor
in a cycle and collecting mid-statement.

_wire_lock is now a small wrapper that tracks owner and depth, so
held_by_current_thread answers the question the finalizer was actually
asking. Same-thread GC defers to the cleanup queue exactly as another
thread's would. This also retires the _is_owned() call in
_ensure_transaction, which was reaching into CPython private API for
the same information.

The finalizer's error handling had the stale-cleanup defect too: a
leftover CLOSE draws -267, an OperationalError, which is in WIRE_ERRORS,
so it force-closed a healthy connection over a no-op. Server-reported
errors are told apart from wire failures by their sqlcode, matching
_drain_pending_cleanup.
2026-09-02 00:29:04 -06:00
ac00a25cb7 A session has one statement slot and the driver acted as if it had many
SQLI gives a session a single statement slot. SQ_CLOSE, SQ_RELEASE and
SQ_SFETCH act on whatever statement is current -- none of them names
one. Ordinary use never notices, because a non-scrollable cursor
materializes its rows and releases the statement before returning.

A scrollable cursor holds the slot open on purpose, and two things went
wrong there without saying so.

Another statement on the same connection got -285, and the scrollable
cursor was collateral damage: its next fetch came back -267, "the
transaction has been rolled back, all locks released". Two errors,
neither naming the cause, from code that reads as completely ordinary --
iterate a large result set, run a lookup partway through. It is now a
ProgrammingError that explains the constraint, and the scrollable cursor
is left alone. Re-executing the same scrollable cursor is the one caller
allowed past the check, and it now closes its own server-side cursor
first, because it collided with itself too.

The deferred-cleanup queue was flushed before every PDU. But a finalizer
enqueues precisely because it lost the wire lock, meaning another thread
is mid-statement -- so the flush landed inside that thread's own
statement and released it. The victim saw -208 when it landed before the
first fetch, -267 between fetch batches. Flushing only at a statement
boundary is both correct and sufficient: cleanup that misses one
statement is picked up by the next.

A stale queue entry was fatal on top of that. The finalizer enqueues,
the cursor is then closed properly, and the leftover CLOSE draws -267 --
an OperationalError, which is in WIRE_ERRORS, which force-closed a
healthy connection. Server-reported errors are now told apart from wire
failures by whether they carry a sqlcode.

Noted while tracing this: _build_close_pdu and _build_release_pdu write
SQ_ID followed by write_int(opcode), which frames correctly only because
the four bytes happen to land as [statement_id=0][opcode]. JDBC's
sendStatementID writes the real id there, and _read_describe_response
has been parsing it into _statement_id all along without ever using it.
Addressing CLOSE/RELEASE correctly is not enough for multiplexing on its
own -- a second cursor's SFETCH addressed to its own id returns -259 --
so that stays open rather than half-done.
2026-09-02 00:24:09 -06:00
0d8bd57ba9 The last two doors out of a statement didn't release it
A statement that fails is still allocated server-side, and skipping the
RELEASE bricks the connection: the next PREPARE collides with the leaked
one and every later call returns a nonsense error whose offset points
back at the failed SQL. That guard has now been added six times, once
per exit, each after a user found the exit. Two were still open.

The parameterized-SELECT bind drain. The build was guarded and the drain
was not, and the drain is where the interesting failures land: passing a
string where the column is an INT encodes cleanly, so the rejection
comes from the server (-1213, -415) after the guard has been passed. A
wrong-typed parameter is about as common as application mistakes get.

The scrollable-cursor open, which had no guard at all. Worst place to
lack one -- the GC-time finalizer is armed on the line after the drain,
so a failure there left the statement allocated with no fallback of any
kind. SELECT ... FOR UPDATE reaches it: prepares clean, fails at OPEN
with -526.

All six now call one helper instead of hand-rolling the cleanup, which
also fixes a defect in the two copies that had a cursor to close. CLOSE
and RELEASE shared a single contextlib.suppress block, so a CLOSE that
raised skipped the RELEASE -- losing the half that matters, since a
leaked cursor handle is a nuisance and a leaked statement breaks the
next call. Each PDU gets its own suppression now.

All four new integration tests fail against the previous commit.
2026-09-02 00:10:23 -06:00
66120cf6f3 Rows never checked that they consumed their own payload
Fourteen framing bugs shipped before this. Every one was a column
reading the wrong number of bytes, and not one of them raised at the
point of the mistake -- the wrong width produced a plausible value and
corrupted whatever came next, so the damage surfaced in a different
column, a different row, or a different statement entirely.

All fourteen were detectable for free. The payload is a fully extracted
bytes object of known length, so after decoding N columns the offset has
to land exactly on the end. It didn't, and nobody looked.

_row_not_consumed now raises when it doesn't, naming the column shape
and the byte delta, and it is wired into all three decode paths --
readers fast path, legacy chain, and the generated decoder.

Turning it on found two more bugs on the first run:

Smart LOBs were read as a flat 72-byte field. They use the UDT envelope:
149 bytes populated ([ind=0][len=144][144 hex chars]), 5 bytes NULL. We
consumed 72 and left 77 behind, so a BLOB or CLOB anywhere but the final
column position shifted every column after it. The 144 bytes are the
72-byte locator hex-encoded, which means BlobLocator.raw had never held
a locator, only the first half of the hex text -- unnoticed because
read_blob_column resolves through the server's lotofile and never reads
the locator.

NULL composite UDTs skipped their length field, byte for byte the
LVARCHAR NULL bug in a branch twelve lines away.

Both are the same [indicator][int32 length][data] envelope, hand-written
a third and fourth time. _read_udt_envelope is now the only copy, and it
rejects a negative length instead of rewinding the offset into bytes it
already decoded.
2026-09-02 00:04:26 -06:00
2e30ceacfb Regenerate uv.lock for 2026.09.01 2026-09-01 22:32:33 -06:00
2896f7e93e TLS traffic fuzzed; no bugs found (2026.09.01)
Tests and docs only, no behaviour change. TLS was the last untested
surface and it came back clean.

Previous coverage stopped at the handshake; everything after it ran only
over plain sockets. That gap mattered because SSLSocket.recv is not
socket.recv: it returns at most one TLS record's worth of plaintext
however much you ask for, can return fewer bytes than are available, and
plaintext buffered inside the SSL object is invisible to the OS. The
Phase 39 buffered reader asks for up to 64 KB per call and loops until
satisfied — that loop had never been pressured the same way.

tests/test_tls_traffic.py runs real SQLI through a TLS-terminating
proxy: the framing-bug types end to end, payloads at 1/4096/16383/16384/
16385/32000 bytes (straddling the ~16 KB record boundary), 500- and
5000-row bulk fetches, error recovery, concurrent TLS sessions, and
three negative cases — TLS client against a plaintext port, plaintext
client against a TLS port (must raise, not hang), and a verifying
context rejecting a self-signed cert.

All clean on all three servers. The buffered reader handles SSLSocket
semantics correctly.

Scope, stated plainly: the proxy supplies the TLS half, so this covers
the driver's TLS path — the half we own. It does NOT cover IBM's
server-side TLS listener. Setting one up on the developer-edition image
was attempted and abandoned: Informix 15 wants a PKCS#12 keystore
(onkstash takes a .p12, not the older CMS .kdb) and the engine kept
rejecting the stash with GSK_ERROR_BAD_KEYFILE_PASSWORD even with a
keystore GSKit itself could open. That half is IBM's code; everything
below ssl.wrap_socket is identical either way.

414/414 integration on 15, 14.10 and 12.10 (was 399).

Every surface is now fuzzed: type framing, fetch batching, error
recovery, cursor lifecycle, threads, pooling, transactions, async
cancellation, executemany partial failure, scrollable cursors, smart
LOBs, and TLS.
2026-09-01 22:32:10 -06:00
488773f078 Regenerate uv.lock for 2026.08.31.3 2026-08-31 19:13:32 -06:00
ac9075ca9a An encoding failure inside executemany leaked the statement (2026.08.31.3)
Last of the untested surface: pipelined executemany, scrollable cursors,
and smart LOBs. One bug, in the first.

executemany builds all its BIND+EXECUTE PDUs after the PREPARE and
before anything is drained — that batching is what makes the pipeline
fast. If encoding a row raises there (a value the connection's codec
cannot represent), the exception escaped without sending the RELEASE,
leaking the prepared statement. The next PREPARE collided with it and
every later call on the connection failed with an error pointing at the
previous SQL.

Same failure as 2026.08.31.2, in a sibling path.
_execute_dml_with_params has guarded this exact case for the single-row
path for a long time; the pipelined path was never given the same
treatment. That is the recurring shape of these bugs: a hazard
understood in one place and not carried to the code beside it.

What the fuzzer found sound, which is the larger part of the result:

  executemany constraint failures — duplicate-key and NOT NULL at
  first/mid/last of batches from 2 to 1000 rows all recover, and
  COUNT(*) always agrees with a full fetch, so the drain-N-responses
  invariant holds under partial failure.

  Scrollable cursors — fetch_first/last/prior/relative/absolute correct
  at 0, 1, 2, 5, 50, 300 rows, including off both ends (None, not a wrap
  or a crash) and a full forward walk after arbitrary positioning.
  Twenty abandoned scroll cursors leak nothing.

  Smart LOBs — round-trip at 0, 1, 255, 256, 1023, 1024, 4095, 4096,
  65535, 65536 bytes, straddling the 4096-byte SQ_FILE chunk and the
  64K mark, plus recovery from failed reads.

Still not fuzzed, stated plainly: TLS is handshake-tested against a
self-signed local socket, not a real Informix TLS listener (that needs
server-side keystore + onconfig SSL setup absent from the test
containers). The SQLI layer above the socket is identical either way.

399/399 integration on 15, 14.10 and 12.10 (was 356).
2026-08-31 19:11:05 -06:00
3d19ca0f1b Regenerate uv.lock for 2026.08.31.2 2026-08-31 17:17:30 -06:00
4af1c90c6f Failed statements bricked the connection; cancelled acquires starved the
pool (2026.08.31.2)

Two more found by fuzzing rather than by users, outside the type system.
Both are ordinary-path bugs a happy-path test cannot reach.

A failed statement was never released. Successful DML sent PREPARE ->
EXECUTE -> RELEASE; failing DML sent PREPARE -> EXECUTE and stopped. The
statement stayed allocated server-side, collided with the next PREPARE,
and every subsequent call on that connection returned a nonsense error
(-255 "Not in transaction" under autocommit, -285 otherwise) whose
offset pointed back at the FAILED sql rather than the new statement.

In practice: one duplicate-key violation bricked the connection. An
INSERT tripping a unique constraint is about the most routine error an
application can hit — every insert-if-not-exists pattern makes them —
and afterwards nothing on that connection worked.

The docstring on the parameterised path already described this hazard
and guarded the parameter-ENCODING failure; it never covered EXECUTE
itself failing. Fixed in all three paths (DML, parameterised DML, and
SELECT — whose fetch loop had the same gap, and whose GC-time finalizer
only covers scrollable cursors).

Cancelling a pool acquire leaked the connection permanently.
asyncio.to_thread cannot interrupt its worker, so a task cancelled while
blocked in pool.acquire() left the worker to finish and hand back a
connection nobody owned — checked out, never returned, one slot gone per
occurrence until the pool was dead.

Not theoretical for anything serving HTTP: a client disconnect cancels
the request task, and under load those cancellations land precisely
while waiting for a connection. The pool dies one slot at a time, only
under load, and PoolTimeoutError points nowhere near the cause.
acquire() now shields the inner future and returns whatever the worker
produced if the caller went away; add_done_callback fires immediately on
an already-resolved future, so the "worker finished just before the
cancel" race is the same code path rather than a second one.

The harness covered three things the type fuzzer cannot see: fetch
batching (NFETCH is a 4096-BYTE budget, so batch edges move with row
width — counts 0..1025 across three schema widths, every fetch style
required to agree), error recovery (eight error classes, repeated, each
followed by a known-good query on the same cursor and a fresh one), and
concurrency (threads sharing one connection, pooled threads that fail
before releasing, transaction isolation across checkouts, async
cancellation). Every worker reads back a value only it supplied, so a
crossed wire fails on data even when nothing raises.

Two harness false positives worth recording: Informix silently truncates
over-long strings (a literal SQL insert does the same, so the driver
matches the server), and it rejects a bare `?` in a projection — a cast
is required.

356/356 integration on 15, 14.10 and 12.10 (was 326).
2026-08-31 17:13:43 -06:00
4e186bb890 Regenerate uv.lock for 2026.08.31.1 2026-08-31 16:01:34 -06:00
15d75dd052 Type-matrix fuzzer, and the three bugs it found (2026.08.31.1)
Six framing bugs had reached users across three reports. Rather than
wait for a seventh, this adds a harness built to find that class of bug.
It immediately found three more, one of which hangs the connection.

BOOLEAN could not be bound as a parameter at all. _encode_bool emitted
type code 45, which the server does not accept as a bind type — it
simply stopped responding, so any execute() with a bool parameter hung
until the read timeout, or forever without one. Type 41 (how BOOLEAN is
*described* in results) hangs identically; descriptor type and bind type
are not interchangeable. Now binds 't'/'f' as CHAR and lets the server
cast, mirroring Informix's own literal syntax. A hang is worse than
wrong data, and this shipped in every release claiming BOOLEAN support.

Unscaled DECIMAL was read one byte short, shifting every following
column. Width is (precision + (scale & 1) + 3) // 2 per
IfxColumnInfo.adjustedColumnLength; we dropped the `scale & 1` term.
That only matters when precision is even and scale is odd — and an
unscaled DECIMAL(p) reports scale 255. DECIMAL(16) is 10 bytes, not 9.
Verified on the wire for ten DECIMAL/MONEY shapes. The same formula
governs DATETIME and INTERVAL, whose qualifier parity tracks their digit
count, so those agreed by coincidence; all four now share one helper
taken from the reference.

NULL CHAR/NCHAR came back as '', indistinguishable from an empty column,
so `WHERE c IS NULL` disagreed with what the driver returned. The wire
distinguishes them plainly — NULL is a leading 0x00, empty is all
spaces.

The harness (tests/test_type_matrix.py) encodes three principles, each
derived from how a real bug escaped:

  1. Always put a column after the value under test. Every bug so far
     corrupted the NEXT column; a trailing value can be mis-sized
     invisibly. Every case ends in a sentinel, and projections rotate.
  2. Vary the data, not just the type. Branch coverage is worthless if
     no value takes the branch — the LVARCHAR fixture was 'lv value',
     8 chars, even, never NULL, so both broken branches sat unexecuted
     through 247 tests.
  3. Use an oracle our codecs don't share. A Python round-trip cannot
     catch a symmetric encode/decode bug: our DATETIME encoder wrote
     zeros and our decoder read them back in perfect agreement. Asking
     the server to render the stored value breaks that symmetry.

Exhaustive over the corpus plus seeded random multi-type rows, so
failures reproduce.

326/326 integration on 15, 14.10 and 12.10 (was 281). Fuzzer reports
clean over 80-round runs at several seeds and widths on all three.
2026-08-31 15:48:48 -06:00
5c6991efba Regenerate uv.lock for 2026.08.31 2026-08-31 14:32:50 -06:00
9616ddbc0a Fix LVARCHAR tuple framing; DATETIME fractions on bind (2026.08.31)
More data corruption from the same field report as 2026.05.08.2. Anyone
with LVARCHAR columns should upgrade: 2026.08.27 is not safe for them.

Two independent errors in the LVARCHAR envelope, both shifting every
column selected after it.

1. A phantom pad byte. We appended an even-byte pad when the value
   length was odd. There is no pad. Wire capture (12.10), INT8 /
   LVARCHAR / INT8:
     00 01 00 00 07 d1 00 00 00 00   a = INT8 2001
     00                              null indicator
     00 00 00 0b                     length 11
     50 61 63 6b 61 67 65 52 6f 6f 74   "PackageRoot" (odd)
     00 01 00 00 00 0a 00 00 00 00   b = INT8 10, starts immediately

2. A missing length on NULL. We returned as soon as the indicator said
   NULL, leaving its 4-byte length unread. The length belongs to the
   envelope and is always present:
     'odd' -> 00 | 00 00 00 03 | 6f 64 64    8 bytes
     NULL  -> 01 | 00 00 00 00               5 bytes
     ''    -> 00 | 00 00 00 00               5 bytes
   NULL and empty string differ only in the indicator byte.

Reported symptom: INT8 10 decoding as 2560 (the same value shifted one
byte left), strings losing their first character, and IndexError or
"INT8 payload too short" once the drift ran off the payload. The
reporter isolated it by reordering columns in the projection — right
when first, wrong when later. That's the fingerprint of positional
drift and a genuinely good diagnostic.

Missed by 247 tests because the only LVARCHAR fixture was 'lv value':
8 characters, even, never NULL. Neither faulty branch ever ran. Same
gap shape as last time — code paths covered, the data reaching them
not. New tests vary parity (0,1,2,3,11,255,256), cover NULL and empty
separately, always place a column AFTER the LVARCHAR, and rotate the
projection through every position.

Separately, DATETIME lost sub-second precision on INSERT. The encoder
emitted YEAR TO SECOND unconditionally, so binding a datetime with
microseconds into a FRACTION(n) column stored zeros, silently. Reads
were always right, so it only went missing on the way in. Now widens
to FRACTION(5) when microsecond is non-zero and keeps the original
encoding otherwise, so the well-exercised path stays byte-identical.
Binding into a narrower column still truncates server-side.

Also: server_version reported 9.56 for a 12.10 server, which reads like
a client-SDK version. The login response only carries the internal
protocol version, and documenting that didn't make the name less
misleading. server_version now returns the release via one cached
DBINFO query; server_version_internal returns the raw login string.

281/281 integration on 15, 14.10 and 12.10; 123 unit tests.
2026-08-31 14:26:17 -06:00
85d3a8f7d7 Docs site: correct APIs that don't exist, document what does
Auditing the docs site against the actual module surface — prompted by
the same field report as 2026.05.08.2 — turned up a partly fictional API.
Every claim below was checked by calling it, not by reading the source.

Documented but nonexistent, now removed:
  conn.transaction()        AttributeError (neither sync nor async)
  conn.autocommit           never existed; set via connect(autocommit=)
  cursor.lastrowid          never existed
  cursor.read_clob_column   never existed
  cursor.write_clob_column  never existed

A copy-pasteable example in reference/types.md raised TypeError:
IntervalYM takes a single total month count, not (years=, months=).

False descriptions corrected: RowValue was described as "named-tuple-like
with .name-accessible fields" and CollectionValue as "iterable" with
"indexed access". Neither is true — both are opaque wrappers over raw
bytes plus a schema string. The docs now say so and point at SQL
projection as the way to get fields today.

Replacements are the real idioms, taken from passing tests:
  CLOBs        -> write_blob_column(..., clob=True); read returns bytes
  SERIAL value -> SELECT DBINFO('sqlca.sqlerrd1')
  transactions -> commit()/rollback() in try/except

Also added: INT8/SERIAL8/BIGSERIAL to the type table (with why INT8 is
not BIGINT), server_capabilities + server_version, the scrollable-cursor
methods, and the note that server_version reports the internal protocol
version (12.10 says 9.56, 14.10 says 9.59).

Updated the test-count claim to 400+ across three server versions,
replacing "integration tests run against 15.0.1.0.3DE".

Added a checker pass over every conn.*/cursor.* reference in the docs;
the only remaining unresolved names are deliberate mentions of IfxPy
APIs we don't implement. Site builds clean.
2026-08-27 09:29:02 -06:00
f5a539d4a8 Add py.typed marker (PEP 561)
pyproject has carried the "Typing :: Typed" classifier without shipping
the marker file, so type checkers ignored the inline annotations
entirely and the classifier was a false claim. The annotations were
always there; only the marker was missing.

Caught during the pre-publish audit.
2026-08-27 02:51:23 -06:00
aa51ee82e0 Regenerate uv.lock after the distribution rename
The lockfile still carried the pre-rename name and a stale version
(informix-db 2026.5.5.9). Same class of miss as the __version__ bug:
the rename touched pyproject.toml but not everything that records the
distribution name.
2026-08-27 02:49:36 -06:00
36478f02c4 Decode the SQ_PROTOCOLS capability negotiation (2026.08.27)
Closes the last open item from the Informix 12 field report. The driver
hardcodes several wire-framing choices that SQLI actually negotiates.
Those choices are correct on every server we've measured, but "correct
as far as we know" and "checked" are different things, and a framing
mismatch corrupts rows silently.

We were already sending SQ_PROTOCOLS with the same 8-byte client offer
IBM's JDBC driver uses, and discarding the reply. Now we decode it.

New informix_db.ServerCapabilities, reachable from any connection:

    conn.server_capabilities.four_byte_offset
    conn.server_capabilities.varchar_var_len
    conn.server_capabilities.violated_assumptions()   # [] when we agree
    conn.server_version

violated_assumptions() names each place we emit or parse a fixed wire
shape that is actually capability-gated. Non-empty at connect time logs
a warning naming the bit, so an untested server produces a diagnosable
complaint instead of quiet corruption.

Nothing branches on these bits yet — this is observation and validation
only. The hardcoded framing is correct on all three supported servers,
and rewriting working parse paths to be conditional without a server
that needs it trades certainty for risk.

The measurement, and why 12/14/15 are interchangeable:

  15.0.1.0.3       bdbe9ffe7fb7ffef ff
  14.10.FC7W1      bdbe9ffe7fb7ffef f8
  12.10.FC12W1DE   bdbe9ffe7fb7ffef f0
                   ^^^^^^^^^^^^^^^^ identical

The first 64 bits are byte-identical. Those releases don't merely behave
alike, they negotiate exactly the same capability set.

Two details worth recording. The reply is NINE bytes; JDBC's
enhancedProtocolMechanism switches on case 0..7 and drops the ninth, so
its BitSet(64) never sees it — yet that dropped byte is the only part
that differs between releases. And Cap_1 in the login response is not a
server version, it's the client's declared protocol level echoed back,
which is why JDBC tests == 316 rather than >=. The version string there
is the internal one: 12.10 reports 9.56, 14.10 reports 9.59. At the
protocol level both really are 9.x servers.

This also retires isUSVER as a red herring: it's one of six bits JDBC
pre-sets for any non-zero Cap_1, and Java's BitSet.set never clears, so
it is true on every modern server regardless of the mask.

Separately, fixed __version__. The distribution was renamed informix-db
-> informix-driver on 2026-05-08 but __init__ kept looking up the old
name. importlib.metadata needs the distribution name, not the module
name, and the miss fails silently — so every install of the renamed
package has reported "0.0.0+local". It escaped notice because a stale
informix-db distribution lingered in the dev venv and answered the
query. tests/test_package_metadata.py now pins the name against
[project].name; verified it catches the bug by reintroducing it.

247/247 integration on 15, 14.10, and 12.10. 120 unit tests.
2026-08-27 02:18:08 -06:00
783b2bec97 Verify Informix 14.10 and 12.10; make the version matrix reproducible
Closes out the field report that prompted 2026.05.08.2. Full integration
suite, same commit, all green:

  Informix 15.0.1.0.3DE     241 / 241
  Informix 14.10.FC7W1DE    241 / 241
  Informix 12.10.FC12W1DE   241 / 241

Including smart-LOB. The 28-type round-trip produces byte-identical wire
output on all three — same type codes, same encoded lengths, same values.
There is no 12-vs-14-vs-15 wire difference for anything we support.

Testing three versions by hand was tedious and undocumented, which is
part of why it never happened. Now:

  make ifx-legacy-up      12.10 on 9089, 14.10 on 9090, alongside 15 on 9088
  make ifx-legacy-setup   blobspace1 + sbspace1 in both
  make test-matrix        the suite against all three

tests/setup-spaces.sh absorbs the per-image differences that made this
annoying: 12.10/14.10 use a flat INFORMIXDIR while 15 nests a versioned
subdirectory; ONCONFIG is named differently across images; `bash -lc`
wipes INFORMIXDIR and makes every utility fail with a misleading
"Unable to read $INFORMIXDIR (/usr/informix)"; and 12.10 DE ships no
ontape at all (the level-0 archive turns out to be unnecessary anyway).
Without blobspace1/sbspace1, ~21 tests fail with errors that look like
driver bugs and aren't.

The documented workflow was validated from scratch — containers removed,
recreated via compose, spaces created via the script, matrix run — rather
than written up afterward from whichever commands happened to work. The
README claim this replaces was wrong precisely because nobody did that.
2026-08-27 00:36:35 -06:00
87b5b7c354 Fix three tuple-framing bugs: BOOLEAN, NCHAR, INT8/SERIAL8 (2026.05.08.2)
Data corruption affecting every Informix version including 15. Two of the
three silently corrupt columns AFTER the offending one, so the damage
shows up far from its cause.

BOOLEAN: described as UDTFIXED(41) with encoded_length=1, but carries the
full UDT envelope on the wire — [1-byte indicator][4-byte length][data],
six bytes for a one-byte value. We read one byte and left five behind.
Captured payload for (INT, BOOLEAN 't', INT, VARCHAR 'tail'):
  00 01 b2 07 | 00 00 00 00 01 74 | 00 03 64 0e | 04 74 61 69 6c
Before: (111111, b'\x00', 1, '\x00\x03d\x0e\x04tail')
After:  (111111, True, 222222, 'tail')

NCHAR: fixed-width and space-padded like CHAR, but we had it grouped with
the byte-length-prefixed types. NCHAR(10) holding 'nch' read 0x6E ('n') as
a 110-byte length. Silent truncation alone, struct.error with any column
following. NVARCHAR really is length-prefixed and is unchanged; there's a
test guarding both sides now.

INT8/SERIAL8: absent from FIXED_WIDTHS, fell through to the unknown-type
path and surfaced as raw bytes. INT8 is not BIGINT — 10 bytes,
sign-magnitude, halves stored high-last:
  bytes 0-1 sign word (0=NULL, 1=pos, 0xFFFF=neg)
  bytes 2-5 LOW 32 bits, bytes 6-9 HIGH 32 bits
+n and -n share magnitude bytes, so a two's-complement read is wrong for
every negative while looking correct for every positive.

Not a version bug. Reported against Informix 12, but the same 28-type
round-trip against 12.10.FC12W1DE and 15.0.1.0.3DE produced byte-identical
wire output. It read as version-specific only because INT8/SERIAL8 dominate
12-era schemas while our fixtures use BIGINT.

251 tests missed all three because no fixture used these types. Added
tests/test_type_framing.py (20 integration) and tests/test_int8_unit.py
(14 unit, wire vectors from both servers). Each type is tested twice —
alone, and with trailing columns, since only the latter catches desync.

Verified: 271/271 integration on 15; 241/241 non-smart-LOB on 12.10 (the
LOB tests need an sbspace that image lacks); 20/20 framing tests on both.

Also corrected README and _fastpath docstring, which asserted 12.10
compatibility that had never been tested. The claims held up, but they
were guesses when written and cost a user debugging time.
2026-08-26 23:40:39 -06:00
2d83ed7b45 Wire-dump: lift overflow-x: auto to base rule (tablet-portrait fix)
Previously the horizontal-scroll fallback only applied at ≤640px; between
641px and 799px (tablet portrait, narrow desktop 2-col), the hero stayed
2-column so the wire-dump column could still be too narrow for the
~564px hex content, and overflow: hidden silently clipped the right side.

Lifting overflow-x: auto to all widths means: (a) any width where the
column is wider than the hex, content displays normally with no scroll;
(b) any width where the column is narrower, content becomes
horizontally-scrollable inside the dump. y stays hidden to keep the
typed-out animation's unrevealed lines clipped below the fold.
2026-05-08 06:07:43 -06:00
dc7b9bfd94 Mobile: fix hero overflow at ≤640px, shrink wire-dump, hide ASCII column
Root cause: hero's grid items inherited min-width: auto, so the wire-dump's
white-space: pre hex lines (~564px wide content) forced the hero column to
that width, propagating up to the page and causing 193px of horizontal
overflow at narrow viewports.

Fix:
- min-width: 0 on .ifx-hero, .ifx-hero__copy, .ifx-hero__visual (lets
  grid items shrink below content's intrinsic min-width)
- overflow-x: auto on .ifx-wiredump (contains residual hex overflow
  inside the dump, not on the page)
- Font shrunk 0.78rem → 0.62rem on mobile, ASCII column hidden, padding
  tightened — readable hex without horizontal scroll inside the dump
- Eyebrow flex-wrap: wrap so the No-libcrypt suffix wraps cleanly
- Title floor 2rem → 1.75rem at 7.5vw for narrow screens
- Tightened CTA padding, install command word-break: break-all
2026-05-08 06:01:47 -06:00
ab5be68738 Hero CTA: 'GitHub' button text → 'Source' (link already pointed at Gitea) 2026-05-08 05:44:11 -06:00
21c47385ae Prose rebrand: informix-db → informix-driver across docs site
Sweep all backticked + bold + table-cell + heading mentions of the
project's brand to match the PyPI distribution name and the docs domain.
Path references (`cd informix-db`, `git.supported.systems/.../informix-db`)
stay — those reference the actual Gitea repo directory which we did NOT
rename. Same with `import informix_db` (Python module name, separate
from distribution brand).

Also flip GitHub references to Gitea throughout the docs site:
- `github.com/rsp2k/informix-db/blob/main/X` → Gitea `/src/branch/main/X`
- `github.com/rsp2k/informix-db/tree/main/X` → Gitea same path
- `github.com/rsp2k/informix-db` (plain) → Gitea
- Hero "GitHub" CTA button → Gitea source URL
- Social icon: `github` → `seti:git` (generic git icon, not octocat)

Net result: zero stale GitHub references, brand consistency matches what
users `pip install`.
2026-05-08 05:43:07 -06:00
b67e6d008b Homepage frontmatter title: informix-db → informix-driver (browser tab consistency) 2026-05-08 05:01:22 -06:00
073c7ed513 Site header: informix-db → informix-driver
Match what users see when they `pip install informix-driver`. Also fix
two broken social links: PyPI was pointing at the now-404 informix-db
project, GitHub link was pointing at github.com/rsp2k/informix-db
(repo doesn't exist there — source lives on Gitea). Rewired to:

- Source (Gitea): git.supported.systems/warehack.ing/informix-db
- PyPI: pypi.org/project/informix-driver
- editLink: Gitea's _edit/branch/main URL pattern

Logo SVG aria-label + title also updated for accessibility consistency.
2026-05-08 05:00:36 -06:00
9af0a4cec9 Rename docs domain: informix-db.warehack.ing → informix-driver.warehack.ing
Brand-consistency with PyPI distribution name. Old domain stops serving
(no redirect — clean cutover per the rename decision).

- pyproject.toml urls: Homepage + Documentation → new domain
- README badge row + sdist exclude comment
- docs-site .env.example + astro.config.mjs site URL + DEV_DOMAIN default
- Version bumped to 2026.05.08.1 (PEP 440 post-release; PyPI 2026.5.8
  still resolvable but with stale URL — yank or leave per preference)
2026-05-08 04:29:08 -06:00
24feabd21b Rename PyPI distribution: informix-db → informix-driver
PyPI rejected `informix-db` as too similar to legacy `informixdb` (v2.5,
2008). Renamed distribution to `informix-driver`. Import name stays
`informix_db` — same separation Pillow uses with `import PIL`.

Updated:
- pyproject.toml [project].name
- README pip-install command + brief explanation note
- docs-site quickstart, vs-ifxpy, wtf, Hero install commands

First PyPI release: pypi.org/project/informix-driver/2026.5.8/
2026-05-08 04:17:57 -06:00
1582a5295d Prepare 2026.05.08 for first PyPI publish
- Bump version 2026.05.05.12 → 2026.05.08 (CalVer, publish date)
- Expand sdist excludes to **/glob patterns: docs/**, docs-site/**,
  tests/reference/**, tests/benchmarks/.results/** — the trailing-slash
  form was silently passing through subtree contents
- Sanitize tests/benchmarks/baseline.json hostname → PLACEHOLDER
- Rewrite README relative docs/, tests/, Makefile links to absolute
  Gitea URLs (git.supported.systems/warehack.ing/informix-db)
- pyproject urls: Homepage + Documentation → informix-db.warehack.ing,
  Source/Issues/Changelog → Gitea (warehack.ing org is now public)
2026-05-08 04:13:09 -06:00
86070e4688 Add docs-site: Astro + Starlight at informix-db.warehack.ing
22 pages across Diataxis quadrants (start / how-to / reference / explain).
Custom amber-on-charcoal theme, wire-dump hero animation, Supported
Systems footer badge. caddy-docker-proxy deployment with prod + dev
profiles, Makefile with prod/dev/down/logs/local targets.
2026-05-08 03:23:22 -06:00
ad55391bf1 Phase 39: Connection-scoped read-ahead buffer (2026.05.05.12)
Closes the bulk-fetch gap to within ~7-15% of IfxPy. Lever was the
buffer/I/O machinery, not the codec — Phase 37/38 had already brought
the codec close to IfxPy's C path; the remaining gap was ~450k
read_exact calls per 100k-row fetch, each doing its own recv-loop
and bytes.join.

Architecture: IfxSocket owns a connection-scoped bytearray + integer
offset cursor; BufferedSocketReader is a thin parser-view that delegates
buffer-fill to the socket. One recv() per ~64 KB instead of per field.
This is how asyncpg (buffer.pyx) and psycopg3 (pq.PGconn) structure
their read paths.

The buffer MUST be socket-scoped, not reader-scoped: the pipelined-
executemany path (Phase 33) streams N responses back-to-back across
multiple cursor reads, and a per-reader buffer would throw away
pre-fetched bytes when one reader is destroyed. (The first iteration
of this phase tried per-reader and hung on test_executemany_1000_rows.)

A/B vs Phase 38, same harness, warmed cache:
  select_scaling_1000     2.90 ->  1.72 ms  (-41%)
  select_scaling_10000   24.32 -> 16.08 ms  (-34%)
  select_scaling_100000 250.36 -> 168.98 ms (-32%)

Head-to-head vs IfxPy 2.0.7:
  select_scaling_1000     1.05x   (basically tied)
  select_scaling_10000    1.07x
  select_scaling_100000   1.15x

IQR collapsed 9x at 100k (3.6 ms -> 0.4 ms) — fewer recvs means fewer
scheduler/jitter pulses showing up in the measurement.

Default ON. Set IFX_BUFFERED_READER=0 to fall back to the legacy reader
(still tested in CI as the escape hatch). Both paths green: 251/251
integration tests pass on each.
2026-05-08 01:40:54 -06:00
a5e6cf1ae3 Phase 38: exec()-based row decoder codegen (2026.05.05.11)
Generates a specialized row-decoder function per result-set shape via
exec(compile(src, ...)) and inlines the common fixed-width decode bodies
directly into the generated source — closing more of the C-vs-Python
codec gap on bulk fetch.

For SMALLINT/INT/SERIAL/BIGINT/BIGSERIAL/FLOAT/SMFLOAT/DATE the decode
body is inlined ("v0 = _UNPACK_INT(raw)[0]; if v0 == sentinel: v0 = None")
rather than called, eliminating one Python function call per such column
per row. BOOL deliberately left to its canonical decoder (Informix BOOL
is 't'/'T'/1, not bool(byte)).

Real A/B vs Phase 37 (median, integration container):
  select_scaling[100000]    257.66 -> 227.67 ms  (-12%)
  wide_row_select[20]         4.27 ->   3.63 ms  (-15%)
  select_scaling[10000]      25.13 ->  22.58 ms  (-10%)
  wide_row_select[100]       15.17 ->  13.59 ms  (-10%)

Win scales with row count and column count — exactly the codegen
profile expected from per-column inlining.

Generated source is printable via IFX_DEBUG_CODEGEN=1.

Three-tier composition: codegen -> reader-list -> legacy chain;
parse_tuple_payload prefers the codegen'd decoder, falls back to the
Phase 37 readers list, falls back to the legacy branch chain.

All 251 integration tests pass.
2026-05-05 14:19:26 -06:00
83 changed files with 17725 additions and 244 deletions

View File

@ -2,6 +2,563 @@
All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440. All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440.
## 2026.09.03 — A placeholder rewriter that could not see a string literal, and named row access
### Upgrade if you write string literals containing a colon
The driver rewrites `:1` placeholders to the `?` the wire protocol takes. That rewrite was a bare regular expression, and a regular expression cannot see a string literal, so it rewrote the inside of one:
```sql
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
-- stored as 'http://host?/x'
```
Any `HH:MM` time, any URL with a port, any aspect ratio, any `key:value` string. It wrote wrong data and said nothing about it.
Two things made it wider than it looks. It was never opt-in: the rewrite runs whenever a statement has parameters, whatever placeholder style you actually used, so writing `?` everywhere and never touching numeric style gave no protection. And it changed the placeholder *count* while the driver still told the server there were `len(params)` of them, leaving the two disagreeing about how many binds exist.
It is now a single pass that substitutes only outside quotes and comments. The lexical rules are Informix's own, measured against all three servers rather than assumed from standard SQL, and one of them would have been got wrong from habit: a backslash escapes nothing, so `'a\'b'` is an unterminated string and draws `-282`. A scanner written to Postgres reflexes would have desynced there and corrupted everything after it. Block comments do not nest, the first `*/` closes them. Braces are a comment. `::` is stepped over as a unit so a cast can never be read as the start of a placeholder.
An unterminated quote or comment consumes the rest and substitutes nothing further. Under-substituting hands malformed SQL to the server to reject; guessing would corrupt a literal.
This was the last place the driver inferred meaning from SQL text instead of handling it properly.
### Rows that answer to a column name
Requested by a user running this alongside SQL Server, where both `pyodbc` and `mssql-python` hand back rows addressable three ways at once. The argument is readability on a wide projection: `row[11]` tells a reader nothing, and stays correct only until somebody adds a column in the middle.
```python
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur.execute("SELECT tabid, tabname FROM systables")
row = cur.fetchone()
row[0], row["tabname"], row.tabname
```
Set on the connection, so it is one line for an application rather than per query. A cursor can override it. Pools and the async API forward it unchanged.
`Row` subclasses `tuple`, so `row == (1, "x")` is still true and code that treats rows as sequences keeps working. Slices degrade to plain tuples, since a slice has no column map.
**It is opt-in, and here is the number.** On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` is a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript. Defaulting it on would move the published 1.05-1.15x ratio against IfxPy to roughly 1.15-1.25x. That is a reasonable trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is a choice rather than a decision made on your behalf.
Details that were measured rather than assumed. Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`, and the `.lower()` in the workaround people write by hand is already a no-op. Expression columns get server-generated names like `(count(*))` that cannot be attributes, so they are subscript-only. Duplicate names resolve to the first occurrence, matching `pyodbc`. And a column always beats a method of the same name: `tuple` defines `count` and `index`, `Row` adds `keys`, `_asdict` and `_fields`, and any column with one of those names wins, because otherwise it would hand back a bound method in silence. That reserved set is computed from the class rather than hand-listed, which is what caught it: the hand-listed version covered the two `tuple` methods and missed all three of `Row`'s own.
### Verified
**472** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **471** on 12.10.FC12W1DE (one skip, no common table expressions before 14.10), up from 457. Both changes have tests that fail against `2026.09.02`.
## 2026.09.02 — A systematic review, and eleven bugs it found
No new features. This is the result of going back over the driver looking for the *shape* of past bugs rather than for new symptoms, and it turned up more than expected — including two that silently returned wrong data and one where `rollback()` did nothing at all.
### Rows never checked that they consumed their own payload
Fourteen framing bugs had reached users before this release. Every one was the same defect — a column reading the wrong number of bytes — and not one of them raised where the mistake happened. The wrong width produced a plausible value and corrupted whatever came *next*, so the damage always surfaced in a different column, a different row, or a different statement.
All fourteen were detectable for free. The payload is a fully extracted `bytes` of known length, so after decoding N columns the offset has to land exactly on the end. Now it's checked, on every row, in all three decode paths, and a mismatch names the column shape and the byte delta instead of handing back plausible garbage.
Turning it on found two more bugs on the first run:
**Smart LOBs were read as a flat 72-byte field.** They actually use the UDT envelope — 149 bytes when populated, 5 when NULL. The driver consumed 72 and left 77 behind, so a `BLOB` or `CLOB` anywhere but the final column position shifted every column after it. The 144 envelope bytes are the 72-byte locator *hex-encoded*, which means `BlobLocator.raw` had never held a locator, only the first half of the hex text. Nobody noticed because `read_blob_column` resolves LOBs server-side and never reads it.
**NULL composite UDTs skipped their length field**, byte for byte the LVARCHAR NULL bug in a branch twelve lines away. Both are now the one `_read_udt_envelope`, which also rejects a negative length rather than rewinding into bytes it already decoded.
### `rollback()` could silently do nothing
`cursor.execute("BEGIN WORK")` is a reasonable thing to write, and with autocommit on it opened a real transaction that the connection never learned about. `commit()` and `rollback()` are both guarded by that flag, so **`rollback()` returned successfully having sent nothing, and the rows it was asked to discard survived.** The connection then went back to the pool holding an open transaction and its locks, because the pool's cleanup is guarded by the same flag.
With autocommit off it failed instead, and for a sillier reason: the driver's implicit `SQ_BEGIN` fired first, so the caller's `BEGIN WORK` got `-535`, "already in transaction". The driver and the user competing to open the same transaction, and the user losing.
The server labels these statements and always has. Both spellings of all three now update the connection's state, and the implicit begin is skipped when the caller is doing the job themselves.
### Five ordinary query forms failed with a nonsense error
The driver decided whether to open a cursor by checking if the first word of the SQL was `SELECT`. That gets wrong:
```sql
-- a leading comment → -260
/* of any of the three flavours */ → -260
{ that Informix accepts } → -260
(SELECT parenthesized) → -260
WITH cte AS (...) SELECT ... → -260
(SELECT a) UNION (SELECT b) → -260
```
`-260` is "Cursor name already in use", which describes neither the cause nor anything the caller did. It says "cursor" because the driver sent `SQ_EXECUTE` where the server was waiting to open one.
The server reports the statement type in the first field of every DESCRIBE response, and the driver had been parsing it into a metadata dict that nothing read. The decision now uses it. `EXECUTE FUNCTION` gets fixed as a side effect — it used to run down the DML path and discard its return value, and now yields it.
### A scrollable cursor took the connection down with it
Informix gives a session one statement slot. A scrollable cursor holds it open on purpose, so any other statement on that connection got `-285`**and destroyed the scrollable cursor too**, whose next fetch came back `-267`, "the transaction has been rolled back, all locks released". Two unattributable failures from code that reads as entirely ordinary: iterate a large result set, run a lookup partway through.
It is now a `ProgrammingError` that explains the constraint and leaves the cursor alone. Re-executing the *same* scrollable cursor is still allowed and now closes its own server-side cursor first, because it collided with itself as well.
### Statements that failed were sometimes never released
A failed statement stays allocated server-side and collides with the next `PREPARE`, after which every call on that connection returns a nonsense error pointing at the *previous* SQL. That guard existed at six exits and was missing from two, both easy to reach:
- The parameterized-`SELECT` bind drain. Passing a string where the column is an `INT` encodes cleanly, so the rejection comes from the server (`-1213`, `-415`) *after* the guarded step.
- Opening a scrollable cursor — no guard at all, and the worst place to lack one, since the GC-time fallback is armed on the line after.
All six now share one implementation, which also fixed a defect in the copies that had a cursor to close: `CLOSE` and `RELEASE` shared a single suppression, so a failing `CLOSE` skipped the `RELEASE` — losing the half that actually matters.
### Cleanup could land inside somebody else's statement
`SQ_CLOSE` and `SQ_RELEASE` act on whatever statement is current; neither names one. Two consequences.
A cursor finalizer that can't take the wire lock queues its cleanup for the next operation to flush — but that queue was flushed before *every* PDU, and a finalizer queues precisely because another thread is mid-statement. The flush landed inside that thread's own statement and released it (`-208` before the first fetch, `-267` between fetch batches). Flushing only at a statement boundary is both correct and sufficient.
And the finalizer's own lock probe couldn't see its own thread. `RLock.acquire(blocking=False)` grants a reentrant acquire to the owner, so when GC fired on a thread that was mid-statement, the finalizer concluded it had exclusive access and sent `CLOSE`/`RELEASE` into the running query. Refcounting hides this — a dropped cursor is freed at the drop — but a cursor caught in a reference cycle waits for a collection, and cycles are ordinary. Any traceback holding a cursor makes one.
A stale queue entry was separately fatal: the leftover `CLOSE` draws `-267`, an `OperationalError`, which was treated as a dead wire and force-closed a perfectly healthy connection.
### The async layer borrowed the whole process's thread pool
Every blocking call went through `asyncio.to_thread`, which runs on the event loop's default executor — sized `min(32, cpu_count + 4)`, so six threads on a two-CPU container.
`asyncio.to_thread` cannot interrupt a worker, so a cancelled await leaves the thread running the wire call until the read timeout. Cancellation is ordinary in a web app; a client disconnect cancels the request task. Measured: six cancelled calls against a six-worker default executor starve an unrelated `to_thread` indefinitely, and with the executor held, **four concurrent driver queries never ran at all**. Pool concurrency was capped by the same unrelated number — `max_size=20` on a two-CPU box ran six queries at a time.
Each connection now owns one thread. That's the right size rather than a compromise, since the connection serializes every wire operation on its own lock anyway, and it avoids a deadlock a shared pool-sized executor invites.
### Two readers shared one stream without agreeing
`IfxSocket` owns a read-ahead buffer that the buffered reader fills, while the login path and the connection-level drain read straight from the socket without looking at it. Buffered bytes would be skipped, and skipped bytes in a length-framed protocol don't announce themselves.
Nothing triggers it today, because the server sends one response per request. That's a property of the traffic, not of the code — pipelined `executemany` already puts several responses in flight, and the buffer is connection-scoped precisely so read-ahead *can* cross response boundaries. The two paths now agree by construction.
Also here: `fill_recv_buf` believed whatever byte count it was handed, and that count is almost always a length field straight off the wire — a garbage `0x7FFFFFFF` read as a 2 GB request and sat in `recv` until the read timeout. It now refuses above `IFX_MAX_READ_BYTES` (256 MiB default) with an error naming the number.
### Verified
**457** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **456** on 12.10.FC12W1DE (one skip — 12.10 has no CTEs) — up from 414. Every fix has at least one test that fails against the previous release.
## 2026.09.01 — TLS traffic fuzzed; no bugs found
Tests and docs only — no behaviour change. TLS was the last untested surface, and it came back clean.
### Why it needed testing separately
Previous TLS coverage stopped at the handshake. Everything after it ran only over plain sockets, and `SSLSocket.recv` is not `socket.recv`: it returns at most one TLS record's worth of plaintext however much you ask for, it can return fewer bytes than are available, and plaintext buffered inside the SSL object is invisible to the OS. The Phase 39 buffered reader asks for up to 64 KB per call and loops until satisfied — that loop is the thing which has to be right, and nothing in the plain-socket suite put the same pressure on it.
`tests/test_tls_traffic.py` runs real SQLI traffic through a TLS-terminating proxy: the framing-bug types end-to-end, payloads at 1 / 4096 / **16383 / 16384 / 16385** / 32000 bytes (straddling the ~16 KB TLS record boundary), 500- and 5000-row bulk fetches, error recovery, concurrent TLS sessions, and three negative cases — TLS client against a plaintext port, plaintext client against a TLS port (must raise rather than hang), and a verifying context correctly rejecting a self-signed certificate.
**All clean on all three servers.** The buffered reader handles `SSLSocket` semantics correctly.
### Scope, stated plainly
The proxy supplies the TLS half, so this exercises the driver's TLS path — the half we own. It does **not** exercise IBM's server-side TLS listener. Setting one up on the developer-edition image was attempted and abandoned: Informix 15 wants a PKCS#12 keystore (`onkstash` takes a `.p12`, not the older CMS `.kdb`), and the engine kept rejecting the stash with `GSK_ERROR_BAD_KEYFILE_PASSWORD` even with a keystore GSKit itself could open. That half is IBM's code; everything below `ssl.wrap_socket` is identical either way.
### Verified
**414/414** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 399.
With this, every surface has been fuzzed: type framing, fetch batching, error recovery, cursor lifecycle, threads, pooling, transactions, async cancellation, `executemany` partial failure, scrollable cursors, smart LOBs, and TLS.
## 2026.08.31.3 — An encoding failure inside `executemany` leaked the statement
The last of the untested surface: pipelined `executemany`, scrollable cursors, and smart LOBs. One bug, in the first of those.
### The bug
`executemany` builds all its BIND+EXECUTE PDUs *after* the PREPARE and before anything is drained — that batching is what makes the pipeline fast. If encoding a row raises there (a value the connection's codec cannot represent), the exception escaped the whole block **without sending the RELEASE**, leaking the prepared statement. The next PREPARE collided with it and every later call on that connection failed with an error pointing at the previous SQL.
Same failure as `2026.08.31.2`, in a sibling path. `_execute_dml_with_params` has guarded this exact case for the single-row path for a long time; the pipelined path was simply never given the same treatment. That is the recurring shape of these: a hazard understood in one place and not carried across to the code next to it.
### What was already sound
Worth recording, because it's the larger part of the result:
- **`executemany` constraint failures.** Duplicate-key and NOT NULL violations at the first, middle, and last position of batches from 2 to 1000 rows all recover cleanly, and `COUNT(*)` always agrees with a full fetch. The pipelined drain-N-responses invariant holds under partial failure.
- **Scrollable cursors.** `fetch_first` / `fetch_last` / `fetch_prior` / `fetch_relative` / `fetch_absolute` are correct at 0, 1, 2, 5, 50 and 300 rows, including off both ends (`None`, not a wrap or a crash) and a full forward walk after arbitrary positioning. Twenty abandoned scroll cursors leak nothing.
- **Smart LOBs.** Round-trip at 0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535 and 65536 bytes — straddling the 4096-byte `SQ_FILE` chunk and the 64K mark — plus recovery from failed reads.
### Verified
**399/399** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 356.
### Honest note on what is still not fuzzed
TLS is covered by handshake tests against a self-signed local socket, which exercises the transport wrapper but not a real Informix TLS listener (that needs server-side keystore and `onconfig` SSL setup we don't have in the test containers). The SQLI layer above the socket is identical either way and is now heavily fuzzed, so the residual risk is confined to the handshake itself.
## 2026.08.31.2 — A failed statement killed the connection; cancelled acquires starved the pool
Two more found by fuzzing rather than by users, this time outside the type system. Both are ordinary-path bugs that a happy-path test cannot reach.
### A failed statement was never released
Successful DML sent `PREPARE → EXECUTE → RELEASE`. **Failing DML sent `PREPARE → EXECUTE` and stopped.** The prepared statement stayed allocated server-side, collided with the next `PREPARE`, and from then on every call on that connection returned a nonsense error — `-255 "Not in transaction"` under autocommit, `-285` otherwise — whose reported offset pointed back at the *failed* SQL rather than the statement that actually failed.
The practical shape of this: **one duplicate-key violation bricked the connection.** An `INSERT` that trips a unique constraint is about the most routine error an application can hit — every "insert if not exists" pattern produces them — and afterwards nothing on that connection worked again.
The docstring on the parameterised path already described this exact hazard and guarded the *parameter-encoding* failure. It just never covered the case where EXECUTE itself failed. Fixed in all three paths (DML, parameterised DML, and SELECT, whose fetch loop had the same gap and whose GC-time finalizer only covers scrollable cursors).
### Cancelling a pool acquire leaked the connection permanently
`asyncio.to_thread` cannot interrupt its worker. When a task awaiting `pool.acquire()` was cancelled *while the worker was still blocked waiting for a free connection*, the worker went on to succeed and hand back a connection **nobody owned** — checked out, never returned. Each occurrence cost one pool slot until the pool was dead.
This is not a theoretical race for anyone serving HTTP. A client disconnecting cancels the request task, and under load those cancellations land precisely while waiting for a connection. The pool dies one slot at a time, only under load, and the eventual symptom (`PoolTimeoutError`) points nowhere near the cause.
`acquire()` now shields the inner future and, if the caller goes away, returns whatever the worker produced to the pool. `add_done_callback` fires immediately on an already-resolved future, so "the worker finished a moment before the cancellation" is the same code path rather than a separate race.
### What the harness looked for
Three areas the type-matrix fuzzer cannot see:
- **Fetch batching.** `NFETCH` is a 4096-*byte* budget, so rows per batch moves with row width. Row counts from 0 to 1025 across narrow/medium/wide schemas, every fetch style (`fetchall`, `fetchone` loop, `fetchmany` at several sizes, iteration) required to agree exactly.
- **Error recovery.** Eight error classes, repeated, each followed by a known-good query on the same cursor *and* a fresh one. Repetition matters: a leak accumulates.
- **Concurrency.** Threads sharing one connection, pooled threads that deliberately fail before releasing, transaction isolation across pool checkouts, and async cancellation. Every worker reads back a value only it supplied, so a crossed wire fails on *data* even when nothing raises.
### Two harness false positives, recorded because they're easy to re-trip
Informix **silently truncates** over-long strings — a 500-character value into `VARCHAR(8)` stores 8 characters and raises nothing. Verified that a literal SQL insert behaves identically, so the driver matches the server; the test expectation was wrong, not the code.
Informix also **rejects a bare `?` in a projection** (`SELECT ? FROM t` is a syntax error — no type to infer). `SELECT ?::INT FROM t` works. Worth knowing before concluding the driver mishandles parameters.
### Verified
**356/356** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 326.
## 2026.08.31.1 — Three more bugs, found by a fuzzer instead of a user
Six framing bugs had reached users across three reports. Rather than wait for a seventh, this release adds a harness built specifically to find that class of bug — and it immediately found three more, one of which **hangs the connection**.
### What it found
**`BOOLEAN` could not be bound as a parameter at all.** `_encode_bool` emitted type code 45, which the server does not accept as a bind type. It simply stopped responding, so *any* `execute()` with a `bool` parameter hung until the read timeout — or forever, if none was set. Type 41 (how BOOLEAN is *described* in results) hangs identically; the descriptor type and the bind type are not interchangeable. We now bind `'t'`/`'f'` as CHAR and let the server cast, mirroring Informix's own literal syntax.
This one is worth dwelling on: a hang is worse than wrong data, and it had shipped in every release that claimed BOOLEAN support.
**Unscaled `DECIMAL` was read one byte short**, shifting every following column. The width formula is `(precision + (scale & 1) + 3) // 2` per `IfxColumnInfo.adjustedColumnLength`; we had dropped the `scale & 1` term. That only matters when precision is even and scale is odd — and an unscaled `DECIMAL(p)` reports scale **255**. So `DECIMAL(16)` is 10 bytes on the wire, not 9. Verified against the wire for ten DECIMAL/MONEY shapes.
The same formula governs DATETIME and INTERVAL, whose qualifier parity happens to track their digit count, so those agreed by coincidence. All four types now share one helper taken from the reference — one rule beats two that coincide for reasons nobody wrote down.
**NULL `CHAR`/`NCHAR` came back as `''`**, indistinguishable from a genuinely empty column, so `WHERE c IS NULL` disagreed with what the driver returned. The wire distinguishes them plainly:
```
'ab' -> 61 62 20 20 20 20
'' -> 20 20 20 20 20 20 all spaces
NULL -> 00 20 20 20 20 20 leading nul
```
### The harness
`tests/test_type_matrix.py` encodes three principles, each derived from how a real bug escaped:
1. **Always put a column after the value under test.** Every bug so far corrupted the *next* column; a trailing value can be mis-sized with no visible effect. Every case now ends in a sentinel column, and projections are rotated so each type is exercised in each position.
2. **Vary the data, not just the type.** Branch coverage means nothing if no value takes the branch. The LVARCHAR fixture was `'lv value'` — 8 characters, even, never NULL — so both of its broken branches sat unexecuted through 247 tests. The corpus now carries odd/even lengths, empty vs NULL, min/max, negative, scaled vs unscaled.
3. **Use an oracle our own codecs don't share.** A Python round-trip cannot catch a symmetric encode/decode bug: our DATETIME encoder wrote zeros and our decoder read them back in perfect agreement. Asking the *server* to render the stored value as text breaks that symmetry, and that check is what would have caught the DATETIME bug on day one.
It runs exhaustively over the corpus plus seeded random multi-type rows, so failures are reproducible.
### Verified
**326/326** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 281. The fuzzer additionally reports clean over 80-round runs at several seeds and widths on all three servers.
## 2026.08.31 — Fix LVARCHAR tuple framing; DATETIME fractions on bind
More data corruption, from the same field report that produced `2026.05.08.2`. **If your schema has `LVARCHAR` columns, upgrade** — anything selected after one could be wrong, and `2026.08.27` is not safe.
### LVARCHAR shifted every column that followed it
Two independent framing errors in the same envelope:
**A phantom pad byte.** We appended an even-byte pad when the value length was odd. There is no pad. Wire capture on Informix 12.10 for `INT8 / LVARCHAR / INT8`:
```
00 01 00 00 07 d1 00 00 00 00 │ 00 │ 00 00 00 0b │ 50 61 63 6b 61 67 65 52 6f 6f 74 │ 00 01 00 00 00 0a …
a = INT8 2001 │ind │ len = 11 │ "PackageRoot" (11 bytes, odd) │ b = INT8 10 starts HERE
```
**A missing length field on NULL.** We returned as soon as the indicator said NULL, leaving its 4-byte length unread. The length is part of the envelope and is always present:
```
'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes
NULL -> 01 | 00 00 00 00 5 bytes
'' -> 00 | 00 00 00 00 5 bytes
```
NULL and empty string differ only in the indicator byte.
Either error shifted everything downstream. The reported symptom: `INT8` `10` decoding as `2560` — the same value shifted one byte left — strings losing their first character, and wide rows raising `IndexError` or `ValueError: INT8 payload too short` once the drift ran off the end of the payload.
The reporter isolated it by **reordering columns in the projection**: values were right when they came first and wrong when they came later. That's the fingerprint of positional drift, and it's a genuinely good diagnostic technique.
### Why the tests missed it, again
The only LVARCHAR fixture was `'lv value'` — 8 characters, even, never NULL. Neither faulty branch ever executed in 247 tests. Same shape of gap as `2026.05.08.2`: full coverage of the code paths, no coverage of the *data* that reaches them.
New tests vary length parity deliberately (0, 1, 2, 3, 11, 255, 256), cover NULL and empty separately, and always put a column *after* the LVARCHAR — a trailing one can be mis-sized with no visible effect. `test_wide_row_survives_column_reordering` rotates the projection through every position, mirroring how this was found.
### DATETIME lost sub-second precision on INSERT
Found while verifying the above. `_encode_datetime` emitted `YEAR TO SECOND` unconditionally, so binding a `datetime` carrying microseconds into a `DATETIME YEAR TO FRACTION(n)` column stored zeros — silently. Reads were always correct, so the value only went missing on the way in.
It now widens to `YEAR TO FRACTION(5)` when `microsecond` is non-zero and keeps the original encoding otherwise, so the long-exercised path is byte-identical for the common case. Binding into a narrower column still works; Informix converts qualifiers on assignment and truncates server-side. `FRACTION(5)` resolves to 10 µs, so Python's sixth digit is dropped — a limit of the type, now pinned by tests.
### `server_version` reported the protocol version
Also from the field report: `conn.server_version` returned `9.56.FC6` for a 12.10 server, which reads like a client-SDK version. The login response only carries Informix's *internal* protocol version — 12.10 announces itself as 9.56, 14.10 as 9.59 — and documenting that didn't make the name any less misleading.
- `conn.server_version` now returns the release (`…Version 12.10.FC6`). It costs one `DBINFO` query on first access, cached thereafter, and falls back rather than raising if no database is open.
- `conn.server_version_internal` returns the raw login string, free as before.
### Verified
281/281 integration tests on 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE (`make test-matrix`), plus 123 unit tests. 37 of those tests are new.
## 2026.08.27 — Decode the SQ_PROTOCOLS capability negotiation
Closes the last open item from the Informix 12 field report. The driver hardcodes several wire-framing choices that SQLI actually *negotiates*; those choices were correct on every server we'd measured, but "correct as far as we know" and "checked" are different things, and the failure mode for a framing mismatch is silently corrupted rows.
### What changed
We were already sending `SQ_PROTOCOLS` with the same 8-byte client offer IBM's JDBC driver uses — and throwing the server's reply away. Now we decode it.
New `informix_db.ServerCapabilities`, reachable from any connection:
```python
conn = informix_db.connect(...)
caps = conn.server_capabilities
caps.four_byte_offset # describe uses 4-byte string-table/field-index widths
caps.varchar_var_len # VARCHAR is length-prefixed, not width-padded
caps.remove_64k_limit # 4-byte length prefixes in the fast-path
caps.violated_assumptions() # [] when our hardcoded framing matches
conn.server_version # from the login response
```
`violated_assumptions()` is the point of the exercise. It names each place we emit or parse a fixed wire shape that is actually capability-gated, and returns empty when the server agrees. On connect, a non-empty result logs a warning naming the specific bit. A server we've never tested now produces a diagnosable complaint instead of quiet corruption.
Nothing branches on these bits yet — this release is observation and validation only. That's deliberate: the hardcoded framing is correct on all three supported servers, and swapping working code for newly-written conditional code without a server that needs it would add risk for no benefit.
### The measurement
All three servers negotiate the same thing:
| Server | Negotiated mask |
|---|---|
| 15.0.1.0.3 | `bdbe9ffe7fb7ffef` `ff` |
| 14.10.FC7W1 | `bdbe9ffe7fb7ffef` `f8` |
| 12.10.FC12W1DE | `bdbe9ffe7fb7ffef` `f0` |
**The first eight bytes are byte-identical.** That is the root explanation for everything in the 2026.05.08.2 investigation: 12.10, 14.10, and 15 don't merely behave similarly, they negotiate exactly the same 64-bit capability set. `violated_assumptions()` returns empty on all three.
Two details worth recording:
The reply is **nine** bytes, not eight. JDBC's `enhancedProtocolMechanism` dispatches on `case 0..7` and silently discards the ninth, so its `BitSet(64)` never sees it. We decode it, because it is the *only* part of the mask that differs between releases — newer feature flags that this JDBC build predates.
`Cap_1` in the login response is **not** a server version. It's the client's own declared protocol level echoed back, which is why JDBC tests `== 316` rather than `>=`. The actual version string there is the *internal* one: Informix 12.10 reports `9.56`, 14.10 reports `9.59`. At the protocol level both really are 9.x servers, which is why the marketing version jump didn't move the wire format.
This also retires a red herring. `isUSVER` (bit 2) is one of six bits JDBC pre-sets for any non-zero `Cap_1`, and Java's `BitSet.set` never clears — so it is true on every modern server regardless of the mask. It can't explain version-specific behaviour and isn't worth chasing.
### Also fixed: `__version__` was reporting a sentinel
Found incidentally while bumping the version. The distribution was renamed `informix-db` to `informix-driver` on 2026-05-08, but `__init__.py` kept calling `version("informix-db")`. `importlib.metadata` needs the *distribution* name, not the module name, and the miss fails silently — `__version__` degrades to `"0.0.0+local"` rather than raising.
So every install of the renamed package has been reporting `0.0.0+local`. It escaped notice locally because a stale `informix-db` distribution was still present in the dev environment and answered the lookup with its own old version.
Added `tests/test_package_metadata.py`, which asserts the name passed to `importlib.metadata.version()` matches `[project].name` in `pyproject.toml`. Verified it actually catches the bug by reintroducing it.
### Tests
24 unit tests over the captured masks (no server needed) and 6 integration tests that assert live negotiation. The integration ones run on every server in `make test-matrix`, so the assumption check is re-verified on 12.10, 14.10, and 15 each time.
Full suite: **247 / 247** on all three.
## 2026.05.08.2 — Fix three tuple-framing bugs (BOOLEAN, NCHAR, INT8/SERIAL8)
Data-corruption fixes. All three affect **every** Informix version including 15, and two of them silently corrupt columns *after* the offending one. If your schema uses `BOOLEAN`, `NCHAR`, `INT8`, or `SERIAL8`, upgrade.
Reported from the field against Informix 12. The version turned out to be a red herring — see "On the version question" below.
### BOOLEAN corrupted every following column
The server describes `BOOLEAN` as `UDTFIXED` (41) with `encoded_length = 1`, but `encoded_length` is the size of the *value*, not the field. On the wire it carries the standard UDT envelope — `[1-byte null indicator][4-byte length][data]` — six bytes for a one-byte value. We consumed one byte and left five behind, shifting everything downstream.
Captured payload for `(INT, BOOLEAN 't', INT, VARCHAR 'tail')`:
```
00 01 b2 07 | 00 00 00 00 01 74 | 00 03 64 0e | 04 74 61 69 6c
a=111111 | b — 6 bytes | c=222222 | d = [len 4]"tail"
```
Before: `(111111, b'\x00', 1, '\x00\x03d\x0e\x04tail')`. After: `(111111, True, 222222, 'tail')`.
### NCHAR ate its first character, then crashed
`NCHAR` is fixed-width and space-padded, exactly like `CHAR`. We had it grouped with the byte-length-prefixed types, so `NCHAR(10)` holding `'nch'` read `0x6E` (`'n'`) as a 110-byte length and advanced the offset by 110. Single-column selects silently returned `'ch'`; anything with a following column raised `struct.error`.
`NVARCHAR` *is* byte-length-prefixed and is unchanged — there's now a regression test guarding both sides of that distinction.
### INT8 / SERIAL8 had no decoder
They were absent from `FIXED_WIDTHS` and fell through to the unknown-type path, surfacing as raw `bytes`. No desync (the width happened to match `encoded_length`), just a wrong type.
`INT8` is **not** `BIGINT`. It's 10 bytes, sign-magnitude, with the halves stored high-last:
```
bytes 0-1 sign word: 0 = NULL, 1 = positive, 0xFFFF = negative
bytes 2-5 LOW 32 bits, big-endian unsigned
bytes 6-9 HIGH 32 bits, big-endian unsigned
```
`+n` and `-n` have identical magnitude bytes, so decoding this as a two's-complement int64 is wrong for every negative value while looking fine for every positive one.
### On the version question
The report arrived as "Informix 12 mangles result sets." It isn't a version issue. Running the same 28-type round-trip against **12.10.FC12W1DE**, **14.10.FC7W1DE**, and **15.0.1.0.3DE** produced **byte-identical** wire output — same type codes, same encoded lengths, same values. All three were equally broken, and are now equally fixed.
The reason it read as version-specific: `INT8`/`SERIAL8` dominate Informix 12-era schemas (`BIGINT` only arrived in 11.50), while our test fixtures use `BIGINT`. The bugs were always there; older schemas just walk into them far more often.
### Why 251 tests missed this
No fixture used `BOOLEAN`, `NCHAR`, `INT8`, or `SERIAL8`. That's the whole explanation. Coverage of *code paths* was good; coverage of the *type matrix* had holes, and the holes were exactly where the bugs lived.
Added `tests/test_type_framing.py` (20 integration tests) and `tests/test_int8_unit.py` (14 unit tests, wire vectors captured from both servers). Every affected type is now tested twice — once for its own value, once with trailing columns, because the trailing-column case is what catches desync. A single-column test passes while the driver corrupts every real query.
### Verified
Full integration suite, same commit, all green:
| Server | Result |
|---|---|
| 15.0.1.0.3DE | 241 / 241 |
| 14.10.FC7W1DE | 241 / 241 |
| 12.10.FC12W1DE | 241 / 241 |
New framing tests pass 34/34 on all three.
### Server matrix is now reproducible
Testing three versions by hand was tedious and undocumented, which is part of why it never happened. Added:
- `tests/docker-compose.legacy.yml` — 12.10 on 9089 and 14.10 on 9090, coexisting with the primary 15 container on 9088
- `tests/setup-spaces.sh` — creates `blobspace1` + `sbspace1` in any dev container. Handles the layout differences between images (12.10/14.10 use a flat `INFORMIXDIR`, 15 nests a versioned subdirectory; `ONCONFIG` is named differently; 12.10 DE ships no `ontape`). Without these spaces ~21 tests fail with errors that look like driver bugs and aren't.
- `make ifx-legacy-up` / `ifx-legacy-setup` / `ifx-legacy-down` / `ifx-spaces`
- `make test-matrix` — the suite against all three versions
The whole documented workflow was validated from scratch (containers destroyed, recreated via compose, spaces created via the script, matrix run) rather than written up after the fact from commands that happened to work.
### Also
`README.md` and `_fastpath.py` claimed 12.10 compatibility that had never been tested. The claims happened to be correct, but they were guesses when written and cost a user debugging time. Both now state what was measured, on which image, on what date.
## 2026.05.05.12 — Phase 39: Connection-scoped read-ahead buffer
Closes the C-vs-Python bulk-fetch gap to within ~7-15% of IfxPy. The lever was the buffer/I/O machinery, not the codec — Phase 37/38 had already brought the codec to within ~25% of IfxPy's C path; the remaining gap was 450k+ ``read_exact`` calls per 100k-row fetch, each doing its own ``recv``-loop and ``bytes.join``.
### What changed
`src/informix_db/_socket.py`:
- ``IfxSocket`` now owns a connection-scoped read-ahead buffer (``_recv_buf`` bytearray + ``_recv_pos`` integer cursor) and exposes ``fill_recv_buf(need)`` to top it up from the socket.
- One ``recv()`` per ~64 KB instead of per field. For a 100k-row SELECT, syscall count drops from ~450k to a few hundred.
- Compaction: when ``_recv_pos > recv_size``, the consumed prefix is sliced off in place. Memory is bounded at roughly ``2 * recv_size`` for the connection's lifetime.
`src/informix_db/_protocol.py`:
- New ``BufferedSocketReader(IfxStreamReader)`` — a thin parser-view over the ``IfxSocket``'s persistent buffer. Each method delegates the buffer-fill to the socket, then reads via ``struct.unpack_from(buf, offset)`` directly out of the bytearray (avoiding the intermediate slice the legacy reader created).
`src/informix_db/cursors.py`:
- ``_make_socket_reader(sock)`` chooses ``BufferedSocketReader`` (default) or the legacy ``_SocketReader`` based on ``IFX_BUFFERED_READER``. Set ``IFX_BUFFERED_READER=0`` to fall back to the legacy reader.
- The four cursor sites that build a reader now call the helper.
### Why the buffer is on `IfxSocket`, not on the reader
The first iteration of Phase 39 put the bytearray on the reader. That hung on `test_executemany_1000_rows` — the pipelined-executemany path (Phase 33) streams N responses back-to-back across multiple cursor reads, and a per-reader buffer threw away pre-fetched bytes when one reader was destroyed and the next was created. The next ``recv()`` then blocked waiting for bytes that had already been consumed.
Moving the buffer to ``IfxSocket`` mirrors how `asyncpg` (`buffer.pyx` lives on the protocol object) and `psycopg3` (`pq.PGconn`) structure their read paths. The reader is a short-lived view; the buffer outlives it.
### Performance
A/B from the same harness, same Docker container, same ``p34_select`` table, warmed cache:
| Workload | Phase 38 | Phase 39 | Δ |
|---|---:|---:|---:|
| `select_scaling_1000` | 2.901 ms | **1.716 ms** | **-41%** |
| `select_scaling_10000` | 24.317 ms | **16.084 ms** | **-34%** |
| `select_scaling_100000` | 250.363 ms | **168.982 ms** | **-32%** |
Against IfxPy 2.0.7 (C-bound, ODBC) on the same workloads, head-to-head:
| Workload | IfxPy | informix-db Phase 39 | Ratio |
|---|---:|---:|---:|
| `select_scaling_1000` | 1.637 ms | 1.716 ms | **1.05×** |
| `select_scaling_10000` | 15.073 ms | 16.084 ms | **1.07×** |
| `select_scaling_100000` | 147.361 ms | 168.982 ms | **1.15×** |
The bulk-fetch gap that was 2.4× before Phase 37 is now within 5-15% of IfxPy. With IfxPy's measurement IQR at ~21% on the 100k workload (Docker→host loopback noise) vs our IQR at ~0.2%, the headline ratio is partly noise — the real gap is plausibly 1.05-1.2×.
A pure-Python driver running within ~10% of an ODBC/C-bound driver on bulk fetch is, as far as I'm aware, unprecedented on the SQLI protocol.
### Tests
All 251 integration tests pass with the new default reader. The legacy reader is exercised via ``IFX_BUFFERED_READER=0`` (also passes 251/251). Both unit suites pass.
### Migration
No code changes required. The default reader switches at version bump. To opt out:
```bash
IFX_BUFFERED_READER=0 python myapp.py
```
If you find a wire shape the buffered reader doesn't handle, please file an issue with the ``IFX_BUFFERED_READER=0`` workaround as the immediate mitigation.
## 2026.05.05.11 — Phase 38: `exec()`-based row-decoder codegen
Closes more of the C-vs-Python codec gap on bulk fetch by emitting a specialized row decoder per result-set shape via `exec(compile(src, ...))` and inlining the common fixed-width decode bodies directly into the generated source. This is the lever flagged in the Phase 37 changelog as "the next lever for materially closing the gap."
### What changed
`src/informix_db/_resultset.py`:
- New `compile_row_decoder(readers, columns)` builds a Python source string per result-set shape and compiles it via `exec()`. The generated function has signature `parse_row(payload, offset, encoding) -> tuple` and contains zero loops — every column is handled by inline straight-line code.
- For the common fixed-width types (`SMALLINT`, `INT`, `SERIAL`, `BIGINT`, `BIGSERIAL`, `FLOAT`, `SMFLOAT`, `DATE`), the decoder body is **inlined** rather than called: `v0 = _UNPACK_INT(raw)[0]; if v0 == -2147483648: v0 = None`. That eliminates one Python function call per such column per row — the actual physics behind the speedup.
- `BOOL` deliberately left to its canonical decoder. Inlining `bool(raw[0])` would silently accept `'f'` (102, truthy) as `True` — semantic drift.
- `parse_tuple_payload` accepts an optional `row_decoder=` parameter. When provided, the entire hot loop is bypassed: `return row_decoder(payload, 0, encoding)`.
- The generated source is printable via `IFX_DEBUG_CODEGEN=1` for inspection.
`src/informix_db/cursors.py`:
- After `parse_describe`, the cursor compiles **both** the Phase 37 reader-list AND the Phase 38 row decoder. `parse_tuple_payload` prefers the codegen'd decoder; if codegen returns `None` (unsupported shape), the readers-list dispatch handles it; if both are `None`, the legacy branch chain runs.
### Performance
Real numbers from the integration container, median of 10+ rounds, A/B against Phase 37 (stash → bench → unstash → bench, same Docker container, same load):
| Benchmark | Phase 37 | Phase 38 | Δ |
|---|---:|---:|---:|
| `select_scaling[1000]` | 2.74 ms | 2.62 ms | -4% |
| `select_scaling[10000]` | 25.13 ms | 22.58 ms | **-10%** |
| `select_scaling[100000]` | 257.66 ms | 227.67 ms | **-12%** |
| `select_type_mix_1000_rows` | 4.57 ms | 4.32 ms | -5% |
| `wide_row_select[5]` | 2.28 ms | 2.05 ms | **-10%** |
| `wide_row_select[20]` | 4.27 ms | 3.63 ms | **-15%** |
| `wide_row_select[50]` | 8.10 ms | 7.19 ms | -11% |
| `wide_row_select[100]` | 15.17 ms | 13.59 ms | **-10%** |
**The win scales with both row count and column count** — exactly the codegen profile we'd expect from per-column inlining. At small row counts the one-time `exec(compile(...))` cost dilutes the per-row win; at 100k rows it's invisible.
### Architectural note
This is conceptually the same step `psycopg3`'s C-mode and `asyncpg` (Cython) take, except we stay 100% pure-Python. We don't compile to native code; we compile to specialized Python bytecode via `exec()`. CPython's bytecode interpreter is remarkably efficient on straight-line code with local variables — the codegen win comes from removing dispatch and function-call overhead, not from native execution.
The three-tier composition stays clean:
1. **Codegen** (`row_decoder`) — fastest path, fires on common shapes
2. **Reader list** (`readers`) — fallback when codegen rejects a shape
3. **Legacy branch chain** — fallback for the no-readers case
### Tests
All 251 integration tests still pass. The codegen output was verified against `IFX_DEBUG_CODEGEN=1` for a 9-column mixed-type shape: SMALLINT/INT/BIGINT/FLOAT/SMFLOAT/DATE/BOOL/CHAR/VARCHAR. All inline NULL-sentinel checks correct; CHAR/VARCHAR fall through to the registered decoder via the globals dict (`_D{i}`). No new test code; the integration suite + benchmark suite are the regression test.
### Honest assessment
Combined with Phase 37 (per-column reader strategy), bulk fetch is now ~20-25% faster than Phase 36 in the worst-affected workloads. The IfxPy gap on `select_scaling[100000]` shrinks from ~2.2× to ~2.0×. Pure-Python finally costs roughly **2× C** for bulk fetch — close enough that the deployment win (no CSDK, no JVM) starts to outweigh the perf cost for most users.
Further codegen wins are possible (inlining DATETIME/INTERVAL, batch-decoding all rows of a payload in one call) but with diminishing returns. The remaining gap is dominated by socket I/O and the SQLI protocol's chatty per-row framing — protocol-level work, not codec work.
## 2026.05.05.10 — Phase 37: Pre-baked per-column reader strategy ## 2026.05.05.10 — Phase 37: Pre-baked per-column reader strategy
Closes some of the C-vs-Python codec gap on bulk fetch by moving per-column dispatch decisions from row time to `parse_describe` time. Same idea as psycopg3's pure-Python loader-cache pattern. Closes some of the C-vs-Python codec gap on bulk fetch by moving per-column dispatch decisions from row time to `parse_describe` time. Same idea as psycopg3's pure-Python loader-cache pattern.

View File

@ -79,6 +79,37 @@ ifx-status: ## Check container health and listener readiness
@docker ps --filter name=$(IFX_CONTAINER) --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}' @docker ps --filter name=$(IFX_CONTAINER) --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'
@nc -zv 127.0.0.1 9088 2>&1 | head -1 @nc -zv 127.0.0.1 9088 2>&1 | head -1
ifx-spaces: ## Create blobspace1 + sbspace1 in the Informix 15 container (needed for LOB tests)
tests/setup-spaces.sh $(IFX_CONTAINER)
# ----------------------------------------------------------------------------
# Server-compatibility matrix: Informix 12.10 / 14.10 alongside 15
# ----------------------------------------------------------------------------
# The legacy containers run on 9089 / 9090 so they coexist with the primary
# 15 container on 9088. The integration suite targets whichever server
# IFX_PORT names, so the matrix is just the same suite three times.
ifx-legacy-up: ## Start Informix 12.10 (9089) and 14.10 (9090) alongside 15
docker compose -f tests/docker-compose.legacy.yml up -d
@echo " 12.10 -> 127.0.0.1:9089"
@echo " 14.10 -> 127.0.0.1:9090"
@echo " Engine init takes ~1-3 min on first start; then: make ifx-legacy-setup"
ifx-legacy-setup: ## Create blobspace1 + sbspace1 in both legacy containers
tests/setup-spaces.sh informix-db-test-1210
tests/setup-spaces.sh informix-db-test-1410
ifx-legacy-down: ## Stop and remove the legacy containers
docker compose -f tests/docker-compose.legacy.yml down
test-matrix: ## Run the integration suite against 15 (9088), 14.10 (9090), and 12.10 (9089)
@echo "=== Informix 15 (9088) ==="
@IFX_PORT=9088 uv run pytest -m "integration and not benchmark" -q --no-header
@echo "=== Informix 14.10 (9090) ==="
@IFX_PORT=9090 uv run pytest -m "integration and not benchmark" -q --no-header
@echo "=== Informix 12.10 (9089) ==="
@IFX_PORT=9089 uv run pytest -m "integration and not benchmark" -q --no-header
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------
# Phase 0 spike: re-capture wire traffic against the dev container # Phase 0 spike: re-capture wire traffic against the dev container
# ---------------------------------------------------------------------------- # ----------------------------------------------------------------------------

View File

@ -1,14 +1,16 @@
# informix-db # informix-driver
Pure-Python driver for IBM Informix IDS, speaking the SQLI wire protocol over raw sockets. **No IBM Client SDK. No JVM. No native libraries.** PEP 249 compliant; sync + async APIs; built-in connection pool; TLS support. Pure-Python driver for IBM Informix IDS, speaking the SQLI wire protocol over raw sockets. **No IBM Client SDK. No JVM. No native libraries.** PEP 249 compliant; sync + async APIs; built-in connection pool; TLS support.
**Docs:** [informix-driver.warehack.ing](https://informix-driver.warehack.ing) · **Source:** [git.supported.systems/warehack.ing/informix-db](https://git.supported.systems/warehack.ing/informix-db) · **PyPI:** [informix-driver](https://pypi.org/project/informix-driver/)
To our knowledge this is the **first pure-socket Informix driver in any language** — every other Informix driver (`IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix`) wraps either IBM's CSDK or the JDBC JAR. To our knowledge this is the **first pure-socket Informix driver in any language** — every other Informix driver (`IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix`) wraps either IBM's CSDK or the JDBC JAR.
```bash ```bash
pip install informix-db pip install informix-driver
``` ```
Requires Python ≥ 3.10. Imports as `informix_db` (the distribution name is `informix-driver` because the legacy `informixdb` package on PyPI from 2008 reserves close-by names — same separation Pillow uses with `import PIL`). Requires Python ≥ 3.10.
## Status ## Status
@ -25,7 +27,7 @@ Requires Python ≥ 3.10.
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test. **0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
**Test coverage:** 300+ tests across unit / integration / benchmark suites. Integration tests run against the official IBM Informix Developer Edition Docker image (15.0.1.0.3DE). **Test coverage:** 400+ tests across unit / integration / benchmark suites. The integration suite passes 414/414 against **each** of Informix 12.10, 14.10, and 15 — `make test-matrix` runs all three.
## Quick start ## Quick start
@ -100,10 +102,11 @@ Informix uses dedicated TLS-enabled listener ports (configured server-side in `s
| SQL type | Python type | | SQL type | Python type |
|---|---| |---|---|
| `SMALLINT` / `INT` / `BIGINT` / `SERIAL` | `int` | | `SMALLINT` / `INT` / `SERIAL` / `BIGINT` / `BIGSERIAL` | `int` |
| `INT8` / `SERIAL8` | `int` (legacy 64-bit — a different wire format from `BIGINT`, not an alias) |
| `FLOAT` / `SMALLFLOAT` | `float` | | `FLOAT` / `SMALLFLOAT` | `float` |
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | | `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` |
| `CHAR` / `VARCHAR` / `NCHAR` / `NVCHAR` / `LVARCHAR` | `str` | | `CHAR` / `NCHAR` (fixed width) · `VARCHAR` / `NVCHAR` / `LVARCHAR` (variable) | `str` |
| `BOOLEAN` | `bool` | | `BOOLEAN` | `bool` |
| `DATE` | `datetime.date` | | `DATE` | `datetime.date` |
| `DATETIME YEAR TO ...` | `datetime.datetime` / `datetime.time` / `datetime.date` | | `DATETIME YEAR TO ...` | `datetime.datetime` / `datetime.time` / `datetime.date` |
@ -131,7 +134,7 @@ cur.write_blob_column(
) )
``` ```
Both work end-to-end in pure Python via the `lotofile` / `filetoblob` server functions intercepted at the `SQ_FILE` (98) wire-protocol level — no native machinery anywhere in the thread of execution. See [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) §1011 for the architecture pivot that made this possible. Both work end-to-end in pure Python via the `lotofile` / `filetoblob` server functions intercepted at the `SQ_FILE` (98) wire-protocol level — no native machinery anywhere in the thread of execution. See [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) §1011 for the architecture pivot that made this possible.
## Direct stored-procedure invocation (fast-path) ## Direct stored-procedure invocation (fast-path)
@ -147,9 +150,46 @@ The fast-path RPC (`SQ_FPROUTINE` / `SQ_EXFPROUTINE`) bypasses PREPARE → EXECU
## Server compatibility ## Server compatibility
Tested against IBM Informix Dynamic Server **15.0.1.0.3DE** (the official `icr.io/informix/informix-developer-database` Docker image). The wire protocol is stable across modern Informix versions; should work against 12.10+ unmodified. All three tested against the official IBM developer-edition Docker images, full integration suite, same commit:
For features that need server-side configuration (smart-LOBs, logged transactions), see [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md): | Server | Image | Integration suite |
|---|---|---|
| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **414 / 414** |
| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
Reproduce the whole matrix:
```bash
make ifx-up # Informix 15 on 9088
make ifx-legacy-up # 12.10 on 9089, 14.10 on 9090
make ifx-legacy-setup # blobspace1 + sbspace1 (smart-LOB tests need these)
make test-matrix # the suite against all three
```
The suite targets whichever server `IFX_PORT` names, so a single version is just `IFX_PORT=9089 uv run pytest -m "integration and not benchmark"`.
Beyond the suite passing, a 28-type round-trip (every scalar type we support, including the fiddly ones — `INT8`, `NCHAR`, `BOOLEAN`, `DATETIME YEAR TO FRACTION(5)`, `INTERVAL`, `DECIMAL`, `MONEY`, `LVARCHAR`) produces **byte-identical wire output** on all three versions: same type codes, same encoded lengths, same values.
Earlier releases of this README claimed the protocol was "stable across modern versions" and should "work against 12.10+ unmodified." That claim turned out to be correct, but it was written before anyone had run it against 12.10, and a user lost debugging time to a version-compatibility problem that didn't exist. It's measured now. Apologies for the earlier guess.
### Capability negotiation
SQLI settles some wire framing through a capability exchange (`SQ_PROTOCOLS`) rather than fixing it by version. This driver decodes that exchange and checks it against the framing it emits:
```python
conn = informix_db.connect(...)
conn.server_version # release, e.g. '…Version 12.10.FC6' (one cached query)
conn.server_version_internal # raw login string; 12.10 announces itself as 9.56
conn.server_capabilities.four_byte_offset # True
conn.server_capabilities.violated_assumptions() # [] — hardcoded framing matches
```
All three servers above negotiate an identical 64-bit capability set, and `violated_assumptions()` is empty on each. If it ever returns something, the driver logs a warning naming the specific bit at connect time — a framing mismatch otherwise shows up as silently corrupted rows, which is a miserable thing to debug.
The framing itself is still hardcoded rather than driven by those bits. That's a deliberate limit: it's correct everywhere we've measured, and rewriting working parse paths to be conditional without a server that needs it would trade certainty for risk. If you hit corrupted rows on a server outside that table, check `violated_assumptions()` first, then please open an issue with your server version, the capability mask, and the `cursor.description` of the offending query.
For features that need server-side configuration (smart-LOBs, logged transactions), see [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md):
- Phase 7 — logged-DB transactions - Phase 7 — logged-DB transactions
- Phase 8 — BYTE/TEXT (needs blobspace) - Phase 8 — BYTE/TEXT (needs blobspace)
- Phase 10/11 — BLOB/CLOB (needs sbspace + `SBSPACENAME` config + level-0 archive) - Phase 10/11 — BLOB/CLOB (needs sbspace + `SBSPACENAME` config + level-0 archive)
@ -168,31 +208,31 @@ Single-connection benchmarks against the dev container on loopback:
| Pool acquire + query + release | 295 µs | ~3.4K queries/sec | | Pool acquire + query + release | 295 µs | ~3.4K queries/sec |
| Cold connect (login handshake) | 11 ms | ~90 connections/sec | | Cold connect (login handshake) | 11 ms | ~90 connections/sec |
**Performance gotcha**: `executemany(...)` under `autocommit=True` is **53× slower** than the same call inside a single transaction (server flushes the transaction log per row). For bulk loads, `autocommit=False` (default) + `conn.commit()` at the end. See [`docs/USAGE.md`](docs/USAGE.md) for the full performance tips section. **Performance gotcha**: `executemany(...)` under `autocommit=True` is **53× slower** than the same call inside a single transaction (server flushes the transaction log per row). For bulk loads, `autocommit=False` (default) + `conn.commit()` at the end. See [`docs/USAGE.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/USAGE.md) for the full performance tips section.
### Compared to IfxPy (the C-bound PyPI driver) ### Compared to IfxPy (the C-bound PyPI driver)
Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/) on identical workloads, same Informix server, matched conditions. Using **median + IQR over 10+ rounds** to resist outlier-round noise: Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/) on identical workloads, same Informix server, matched conditions. Using **median + IQR over 10+ rounds** to resist outlier-round noise:
| Benchmark | IfxPy 3.0.5 (C-bound) | `informix-db` (pure Python) | Result | | Benchmark | IfxPy 3.0.5 (C-bound) | `informix-driver` (pure Python) | Result |
|---|---:|---:|---:| |---|---:|---:|---:|
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable | | Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster | | ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
| Cold connect (login handshake) | 11.0 ms | 10.5 ms | comparable | | Cold connect (login handshake) | 11.0 ms | 10.5 ms | comparable |
| **`executemany(1k)` in transaction** | 23.5 ms | 23.2 ms | tied | | **`executemany(1k)` in transaction** | 23.5 ms | 23.2 ms | tied |
| **`executemany(10k)` in transaction** | 259 ms | **161 ms** | **`informix-db` 1.6× faster** | | **`executemany(10k)` in transaction** | 259 ms | **161 ms** | **`informix-driver` 1.6× faster** |
| **`executemany(100k)` in transaction** | 2376 ms | **1487 ms** | **`informix-db` 1.6× faster** | | **`executemany(100k)` in transaction** | 2376 ms | **1487 ms** | **`informix-driver` 1.6× faster** |
| `SELECT` 1k rows | 1.2 ms | 2.7 ms | IfxPy 2.3× faster | | `SELECT` 1k rows | 1.2 ms | 2.7 ms | IfxPy 2.3× faster |
| `SELECT` 10k rows | 11.3 ms | 25.8 ms | IfxPy 2.3× faster | | `SELECT` 10k rows | 11.3 ms | 25.8 ms | IfxPy 2.3× faster |
| `SELECT` 100k rows | 112 ms | 271 ms | IfxPy 2.4× faster | | `SELECT` 100k rows | 112 ms | 271 ms | IfxPy 2.4× faster |
**The honest summary:** **The honest summary:**
- **Bulk-insert workloads: `informix-db` wins 1.6× at scale.** The pipelined `executemany` (Phase 33) sends all N BIND+EXECUTE PDUs before draining responses, eliminating per-row RTT. IfxPy still pays one round-trip per `IfxPy.execute(stmt, tuple)` call. - **Bulk-insert workloads: `informix-driver` wins 1.6× at scale.** The pipelined `executemany` (Phase 33) sends all N BIND+EXECUTE PDUs before draining responses, eliminating per-row RTT. IfxPy still pays one round-trip per `IfxPy.execute(stmt, tuple)` call.
- **Large-fetch workloads: IfxPy wins 2.3× at scale.** Their C-level `fetch_tuple` decoder is genuinely faster than our Python `parse_tuple_payload` (~1.1 µs/row vs ~2.7 µs/row). At 100k rows, that 1.6 µs/row gap accumulates into a 160 ms wall-clock difference. - **Large-fetch workloads: IfxPy wins 2.3× at scale.** Their C-level `fetch_tuple` decoder is genuinely faster than our Python `parse_tuple_payload` (~1.1 µs/row vs ~2.7 µs/row). At 100k rows, that 1.6 µs/row gap accumulates into a 160 ms wall-clock difference.
- **Small queries: comparable.** Both spend ~120 µs waiting for the server; the per-call codec cost is small relative to the round-trip. - **Small queries: comparable.** Both spend ~120 µs waiting for the server; the per-call codec cost is small relative to the round-trip.
**When to prefer `informix-db`:** **When to prefer `informix-driver`:**
- ETL pipelines, log shipping, bulk writes (1.6× faster at scale) - ETL pipelines, log shipping, bulk writes (1.6× faster at scale)
- Containerized / minimal-dependency environments (50 KB wheel vs IfxPy's 92 MB OneDB tarball + libcrypt.so.1 dependency hell) - Containerized / minimal-dependency environments (50 KB wheel vs IfxPy's 92 MB OneDB tarball + libcrypt.so.1 dependency hell)
- Modern Python (works on 3.103.14; IfxPy is broken on Python 3.12+) - Modern Python (works on 3.103.14; IfxPy is broken on Python 3.12+)
@ -204,9 +244,9 @@ Head-to-head benchmarks against [IfxPy](https://pypi.org/project/IfxPy/) on iden
These results are reproducible from `tests/benchmarks/compare/` — the Dockerfile, bench script, and README walk through every step. These results are reproducible from `tests/benchmarks/compare/` — the Dockerfile, bench script, and README walk through every step.
Full methodology, IQR caveats, install gauntlet, and reproduction in [`tests/benchmarks/compare/README.md`](tests/benchmarks/compare/README.md). Full methodology, IQR caveats, install gauntlet, and reproduction in [`tests/benchmarks/compare/README.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare/README.md).
A note on IfxPy's install gauntlet: getting it to run on a modern system requires Python ≤ 3.11, setuptools <58, permissive CFLAGS, manual download of a 92 MB ODBC tarball, four `LD_LIBRARY_PATH` directories, and `libcrypt.so.1` (deprecated 2018, missing on Arch / Fedora 35+ / RHEL 9). `informix-db`'s install: `pip install informix-db`. A note on IfxPy's install gauntlet: getting it to run on a modern system requires Python ≤ 3.11, setuptools <58, permissive CFLAGS, manual download of a 92 MB ODBC tarball, four `LD_LIBRARY_PATH` directories, and `libcrypt.so.1` (deprecated 2018, missing on Arch / Fedora 35+ / RHEL 9). `informix-driver`'s install: `pip install informix-driver`.
## Standards & guarantees ## Standards & guarantees
@ -217,7 +257,7 @@ A note on IfxPy's install gauntlet: getting it to run on a modern system require
## Development ## Development
The full test + lint workflow is in the [Makefile](Makefile). Quick summary: The full test + lint workflow is in the [Makefile](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/Makefile). Quick summary:
```bash ```bash
make test # 77 unit tests (no Docker) make test # 77 unit tests (no Docker)
@ -226,22 +266,22 @@ make bench # benchmark suite
make lint # ruff make lint # ruff
``` ```
For the smart-LOB tests specifically, the dev container needs additional one-time setup (blobspace + sbspace + level-0 archive). See [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) §10 for the `onspaces` / `onmode` / `ontape` commands. For the smart-LOB tests specifically, the dev container needs additional one-time setup (blobspace + sbspace + level-0 archive). See [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) §10 for the `onspaces` / `onmode` / `ontape` commands.
## Documentation ## Documentation
- [**`docs/USAGE.md`**](docs/USAGE.md) — practical recipes: connections, parameter binding, type mapping, transactions, performance tips, scrollable cursors, BLOBs, async, TLS, locale/Unicode, error handling, known limitations - [**`docs/USAGE.md`**](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/USAGE.md) — practical recipes: connections, parameter binding, type mapping, transactions, performance tips, scrollable cursors, BLOBs, async, TLS, locale/Unicode, error handling, known limitations
- [`tests/benchmarks/README.md`](tests/benchmarks/README.md) — performance baselines, headline numbers, how to run regressions - [`tests/benchmarks/README.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/README.md) — performance baselines, headline numbers, how to run regressions
- `CHANGELOG.md` — phase-by-phase release notes - `CHANGELOG.md` — phase-by-phase release notes
## Project history & design rationale ## Project history & design rationale
This driver was built incrementally across 30 phases, each with a focused scope and decision log. The reasoning trail lives in: This driver was built incrementally across 30 phases, each with a focused scope and decision log. The reasoning trail lives in:
- [`docs/PROTOCOL_NOTES.md`](docs/PROTOCOL_NOTES.md) — byte-level SQLI wire-format reference - [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) — byte-level SQLI wire-format reference
- [`docs/JDBC_NOTES.md`](docs/JDBC_NOTES.md) — index into the decompiled IBM JDBC driver, used as a clean-room reference - [`docs/JDBC_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/JDBC_NOTES.md) — index into the decompiled IBM JDBC driver, used as a clean-room reference
- [`docs/DECISION_LOG.md`](docs/DECISION_LOG.md) — phase-by-phase architectural decisions, with the *why* preserved - [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) — phase-by-phase architectural decisions, with the *why* preserved
- [`docs/CAPTURES/`](docs/CAPTURES/) — annotated socat hex-dump captures - [`docs/CAPTURES/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/CAPTURES/) — annotated socat hex-dump captures
Notable architectural pivots documented in the decision log: Notable architectural pivots documented in the decision log:
- **Phase 10/11** (smart-LOB read/write): used `lotofile`/`filetoblob` SQL functions + `SQ_FILE` protocol intercept instead of the heavier `SQ_FPROUTINE` + `SQ_LODATA` stack — ~3x smaller than originally projected - **Phase 10/11** (smart-LOB read/write): used `lotofile`/`filetoblob` SQL functions + `SQ_FILE` protocol intercept instead of the heavier `SQ_FPROUTINE` + `SQ_LODATA` stack — ~3x smaller than originally projected

12
docs-site/.dockerignore Normal file
View File

@ -0,0 +1,12 @@
node_modules
dist
.astro
.env
.env.local
.git
*.log
README.md
.dockerignore
Dockerfile
docker-compose.yml
Makefile

4
docs-site/.env.example Normal file
View File

@ -0,0 +1,4 @@
COMPOSE_PROJECT=informix-db-docs
DOMAIN=informix-driver.warehack.ing
DEV_DOMAIN=informix-driver.l.warehack.ing
MODE=prod

21
docs-site/.gitignore vendored Normal file
View File

@ -0,0 +1,21 @@
# build output
dist/
# generated types
.astro/
# dependencies
node_modules/
# logs
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
# environment variables
.env
.env.production
# macOS-specific files
.DS_Store

20
docs-site/Caddyfile Normal file
View File

@ -0,0 +1,20 @@
:80 {
root * /srv
encode zstd gzip
file_server
@assets path *.css *.js *.woff2 *.woff *.svg *.png *.webp *.jpg *.jpeg *.ico
header @assets Cache-Control "public, max-age=31536000, immutable"
@html path *.html
header @html Cache-Control "public, max-age=300, must-revalidate"
header @html X-Content-Type-Options "nosniff"
header @html X-Frame-Options "SAMEORIGIN"
header @html Referrer-Policy "strict-origin-when-cross-origin"
handle_errors {
@404 expression `{err.status_code} == 404`
rewrite @404 /404.html
file_server
}
}

30
docs-site/Dockerfile Normal file
View File

@ -0,0 +1,30 @@
ARG NODE_VERSION=22-alpine
ARG CADDY_VERSION=2.10-alpine
FROM node:${NODE_VERSION} AS deps
WORKDIR /app
COPY package.json package-lock.json* ./
RUN --mount=type=cache,target=/root/.npm \
npm ci --no-audit --no-fund 2>/dev/null || npm install --no-audit --no-fund
FROM node:${NODE_VERSION} AS builder
WORKDIR /app
ENV ASTRO_TELEMETRY_DISABLED=1 NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build
FROM caddy:${CADDY_VERSION} AS prod
COPY --from=builder /app/dist /srv
COPY Caddyfile /etc/caddy/Caddyfile
EXPOSE 80
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD wget -q --spider http://127.0.0.1/ || exit 1
FROM node:${NODE_VERSION} AS dev
WORKDIR /app
ENV ASTRO_TELEMETRY_DISABLED=1 NODE_ENV=development
COPY --from=deps /app/node_modules ./node_modules
COPY . .
EXPOSE 4321
CMD ["npm", "run", "dev", "--", "--host", "0.0.0.0"]

33
docs-site/Makefile Normal file
View File

@ -0,0 +1,33 @@
.PHONY: prod dev down logs build clean install local
include .env
export
prod:
docker compose --profile prod up -d --build
dev:
docker compose --profile dev up -d --build
down:
docker compose --profile prod --profile dev down
logs:
docker compose logs -f --tail=200
build:
npm run build
clean:
rm -rf dist node_modules .astro
install:
npm install
local:
npm run dev -- --host 127.0.0.1
status:
@echo "Domain: $(DOMAIN)"
@echo "Project: $(COMPOSE_PROJECT)"
@docker compose ps

49
docs-site/README.md Normal file
View File

@ -0,0 +1,49 @@
# Starlight Starter Kit: Basics
[![Built with Starlight](https://astro.badg.es/v2/built-with-starlight/tiny.svg)](https://starlight.astro.build)
```
npm create astro@latest -- --template starlight
```
> 🧑‍🚀 **Seasoned astronaut?** Delete this file. Have fun!
## 🚀 Project Structure
Inside of your Astro + Starlight project, you'll see the following folders and files:
```
.
├── public/
├── src/
│ ├── assets/
│ ├── content/
│ │ └── docs/
│ └── content.config.ts
├── astro.config.mjs
├── package.json
└── tsconfig.json
```
Starlight looks for `.md` or `.mdx` files in the `src/content/docs/` directory. Each file is exposed as a route based on its file name.
Images can be added to `src/assets/` and embedded in Markdown with a relative link.
Static assets, like favicons, can be placed in the `public/` directory.
## 🧞 Commands
All commands are run from the root of the project, from a terminal:
| Command | Action |
| :------------------------ | :----------------------------------------------- |
| `npm install` | Installs dependencies |
| `npm run dev` | Starts local dev server at `localhost:4321` |
| `npm run build` | Build your production site to `./dist/` |
| `npm run preview` | Preview your build locally, before deploying |
| `npm run astro ...` | Run CLI commands like `astro add`, `astro check` |
| `npm run astro -- --help` | Get help using the Astro CLI |
## 👀 Want to learn more?
Check out [Starlights docs](https://starlight.astro.build/), read [the Astro documentation](https://docs.astro.build), or jump into the [Astro Discord server](https://astro.build/chat).

119
docs-site/astro.config.mjs Normal file
View File

@ -0,0 +1,119 @@
// @ts-check
import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';
const DEV_DOMAIN = process.env.DEV_DOMAIN ?? 'informix-driver.l.warehack.ing';
// https://astro.build/config
export default defineConfig({
site: 'https://informix-driver.warehack.ing',
server: { host: '0.0.0.0', port: 4321 },
telemetry: false,
devToolbar: { enabled: false },
vite: {
server: {
host: '0.0.0.0',
hmr: {
host: DEV_DOMAIN,
protocol: 'wss',
clientPort: 443,
},
allowedHosts: [DEV_DOMAIN, '.warehack.ing', 'localhost', '127.0.0.1'],
},
},
integrations: [
starlight({
title: 'informix-driver',
description: 'Pure-Python driver for IBM Informix IDS. No CSDK, no JVM, no native libraries.',
logo: { src: './src/assets/logo.svg', replacesTitle: false },
favicon: '/favicon.svg',
tableOfContents: { minHeadingLevel: 2, maxHeadingLevel: 4 },
lastUpdated: true,
pagination: true,
editLink: {
baseUrl: 'https://git.supported.systems/warehack.ing/informix-db/_edit/branch/main/docs-site/',
},
social: [
{ icon: 'seti:git', label: 'Source (Gitea)', href: 'https://git.supported.systems/warehack.ing/informix-db' },
{ icon: 'seti:python', label: 'PyPI', href: 'https://pypi.org/project/informix-driver/' },
],
customCss: ['./src/styles/theme.css', './src/styles/components.css'],
components: {
Hero: './src/components/Hero.astro',
Footer: './src/components/Footer.astro',
},
expressiveCode: {
themes: ['github-dark', 'github-light'],
styleOverrides: {
borderRadius: '6px',
codeFontFamily: "'IBM Plex Mono', ui-monospace, SFMono-Regular, Menlo, monospace",
},
},
head: [
{
tag: 'link',
attrs: { rel: 'preconnect', href: 'https://rsms.me' },
},
{
tag: 'link',
attrs: {
rel: 'stylesheet',
href: 'https://rsms.me/inter/inter.css',
},
},
{
tag: 'meta',
attrs: { name: 'theme-color', content: '#0e0d0c' },
},
{
tag: 'meta',
attrs: { property: 'og:type', content: 'website' },
},
],
sidebar: [
{
label: 'Start here',
items: [
{ label: 'WTF did you build this for?', slug: 'start/wtf' },
{ label: 'Install & first query', slug: 'start/quickstart' },
{ label: 'Compared to IfxPy', slug: 'start/vs-ifxpy' },
],
},
{
label: 'How-to guides',
items: [
{ label: 'Connect with TLS', slug: 'how-to/tls' },
{ label: 'Use the connection pool', slug: 'how-to/pool' },
{ label: 'Async with FastAPI', slug: 'how-to/async-fastapi' },
{ label: 'Bulk inserts (executemany)', slug: 'how-to/executemany' },
{ label: 'Optimize bulk SELECT', slug: 'how-to/buffered-reader' },
{ label: 'BLOB / CLOB read & write', slug: 'how-to/smart-lobs' },
{ label: 'Migrate from IfxPy', slug: 'how-to/migrate-from-ifxpy' },
{ label: 'Run the dev container', slug: 'how-to/dev-container' },
],
},
{
label: 'Reference',
items: [
{ label: 'API surface', slug: 'reference/api' },
{ label: 'SQL ↔ Python types', slug: 'reference/types' },
{ label: 'Configuration & env flags', slug: 'reference/config' },
{ label: 'Exceptions & error codes', slug: 'reference/exceptions' },
{ label: 'Performance baselines', slug: 'reference/benchmarks' },
],
},
{
label: 'Explanation',
items: [
{ label: 'The SQLI wire protocol', slug: 'explain/sqli-protocol' },
{ label: 'Architecture overview', slug: 'explain/architecture' },
{ label: 'The buffered reader (Phase 39)', slug: 'explain/buffered-reader' },
{ label: 'Async strategy', slug: 'explain/async-strategy' },
{ label: 'Pure-Python tradeoffs', slug: 'explain/pure-python' },
{ label: 'The phase log', slug: 'explain/phase-log' },
],
},
],
}),
],
});

View File

@ -0,0 +1,46 @@
services:
docs:
profiles: ["prod"]
build:
context: .
target: prod
container_name: ${COMPOSE_PROJECT}-prod
restart: unless-stopped
networks:
- caddy
labels:
caddy: ${DOMAIN}
caddy.reverse_proxy: "{{upstreams 80}}"
docs-dev:
profiles: ["dev"]
build:
context: .
target: dev
container_name: ${COMPOSE_PROJECT}-dev
restart: unless-stopped
volumes:
- ./src:/app/src:cached
- ./public:/app/public:cached
- ./astro.config.mjs:/app/astro.config.mjs:cached
- ./tsconfig.json:/app/tsconfig.json:cached
- ./package.json:/app/package.json:cached
environment:
DEV_DOMAIN: ${DEV_DOMAIN}
networks:
- caddy
labels:
caddy: ${DEV_DOMAIN}
caddy.reverse_proxy: "{{upstreams 4321}}"
caddy.reverse_proxy.flush_interval: "-1"
caddy.reverse_proxy.transport: "http"
caddy.reverse_proxy.transport.read_timeout: "0"
caddy.reverse_proxy.transport.write_timeout: "0"
caddy.reverse_proxy.transport.keepalive: "5m"
caddy.reverse_proxy.transport.keepalive_idle_conns: "10"
caddy.reverse_proxy.stream_timeout: "24h"
caddy.reverse_proxy.stream_close_delay: "5s"
networks:
caddy:
external: true

6269
docs-site/package-lock.json generated Normal file

File diff suppressed because it is too large Load Diff

17
docs-site/package.json Normal file
View File

@ -0,0 +1,17 @@
{
"name": "docs-site",
"type": "module",
"version": "0.0.1",
"scripts": {
"dev": "astro dev",
"start": "astro dev",
"build": "astro build",
"preview": "astro preview",
"astro": "astro"
},
"dependencies": {
"@astrojs/starlight": "^0.39.1",
"astro": "^6.2.2",
"sharp": "^0.34.5"
}
}

View File

@ -0,0 +1,6 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 32 32">
<rect width="32" height="32" rx="6" fill="#0e0d0c"/>
<rect x="8" y="11" width="6" height="6" rx="1" fill="#f5a524"/>
<rect x="8" y="19" width="16" height="2" rx="1" fill="#f5a524"/>
<rect x="8" y="23" width="11" height="2" rx="1" fill="#f5a524" opacity="0.55"/>
</svg>

After

Width:  |  Height:  |  Size: 338 B

View File

@ -0,0 +1,66 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 100 75" height="100%" width="100%">
<!-- Gradient Definitions -->
<defs>
<linearGradient id="gradient1" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" stop-color="#60a5fa"></stop>
<stop offset="50%" stop-color="#3b82f6"></stop>
<stop offset="100%" stop-color="#2563eb"></stop>
</linearGradient>
<linearGradient id="gradient2" x1="0%" y1="0%" x2="0%" y2="100%">
<stop offset="0%" stop-color="#93c5fd"></stop>
<stop offset="50%" stop-color="#60a5fa"></stop>
<stop offset="100%" stop-color="#3b82f6"></stop>
</linearGradient>
<linearGradient id="flowGradient" x1="0%" y1="0%" x2="100%" y2="0%">
<stop offset="0%" stop-color="#2563eb"></stop>
<stop offset="50%" stop-color="#60a5fa"></stop>
<stop offset="100%" stop-color="#2563eb"></stop>
</linearGradient>
<pattern id="circuitPattern" patternUnits="userSpaceOnUse" width="12" height="45" patternTransform="scale(1)">
<rect width="12" height="45" fill="url(#gradient1)"></rect>
<path d="M2,5 h8 M2,5 v5 M10,5 v10 M5,15 h5 M5,15 v10 M3,25 h7 M7,25 v10 M3,35 h4" stroke="#dbeafe" stroke-width="0.5" fill="none" opacity="0.7"></path>
<circle cx="2" cy="5" r="1" fill="#dbeafe" opacity="0.7"></circle>
<circle cx="10" cy="5" r="1" fill="#dbeafe" opacity="0.7"></circle>
<circle cx="5" cy="15" r="1" fill="#dbeafe" opacity="0.7"></circle>
<circle cx="3" cy="25" r="1" fill="#dbeafe" opacity="0.7"></circle>
<circle cx="7" cy="35" r="1" fill="#dbeafe" opacity="0.7"></circle>
</pattern>
<pattern id="binaryPattern" patternUnits="userSpaceOnUse" width="12" height="35" patternTransform="scale(1)">
<rect width="12" height="35" fill="#2563eb"></rect>
<text x="3" y="8" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">10</text>
<text x="3" y="14" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">01</text>
<text x="3" y="20" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">11</text>
<text x="3" y="26" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">00</text>
<text x="3" y="32" font-family="monospace" font-size="3" fill="#FFFFFF" opacity="0.5">10</text>
</pattern>
<pattern id="punchCardPattern" patternUnits="userSpaceOnUse" width="12" height="45" patternTransform="scale(1)">
<rect width="12" height="45" fill="#3b82f6"></rect>
<path d="M0,5 h12 M0,10 h12 M0,15 h12 M0,20 h12 M0,25 h12 M0,30 h12 M0,35 h12 M0,40 h12" stroke="#93c5fd" stroke-width="0.2" fill="none"></path>
<circle cx="3" cy="7" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="9" cy="7" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="6" cy="12" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="3" cy="17" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="9" cy="22" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="6" cy="27" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="3" cy="32" r="1" fill="#1e3a8a" opacity="0.9"></circle>
<circle cx="9" cy="37" r="1" fill="#1e3a8a" opacity="0.9"></circle>
</pattern>
</defs>
<!-- Flow lines behind bars -->
<g opacity="0.3">
<path d="M6,50 C20,40 40,55 48,35 C56,50 75,30 90,55" stroke="url(#flowGradient)" stroke-width="1" fill="none"></path>
<path d="M6,60 C30,50 50,40 70,55 C80,45 90,60 90,60" stroke="url(#flowGradient)" stroke-width="1" fill="none"></path>
</g>
<!-- Bar chart graphic - the "towers" -->
<g>
<rect x="0" y="45" width="12" height="25" rx="1" ry="1" fill="url(#binaryPattern)"></rect>
<rect x="14" y="35" width="12" height="35" rx="1" ry="1" fill="#2563eb"></rect>
<rect x="28" y="25" width="12" height="45" rx="1" ry="1" fill="url(#circuitPattern)"></rect>
<rect x="42" y="20" width="12" height="50" rx="1" ry="1" fill="url(#gradient2)"></rect>
<rect x="56" y="25" width="12" height="45" rx="1" ry="1" fill="url(#punchCardPattern)"></rect>
<rect x="70" y="35" width="12" height="35" rx="1" ry="1" fill="url(#circuitPattern)"></rect>
<rect x="84" y="45" width="12" height="25" rx="1" ry="1" fill="#2563eb"></rect>
<!-- Connecting glow -->
<path d="M12,55 L14,55 M26,45 L28,45 M40,40 L42,40 M54,40 L56,40 M82,55 L84,55" stroke="#bfdbfe" stroke-width="0.8" stroke-opacity="0.6"></path>
</g>
</svg>

After

Width:  |  Height:  |  Size: 4.3 KiB

View File

@ -0,0 +1,7 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 28 24" fill="none" role="img" aria-label="informix-driver">
<title>informix-driver</title>
<rect x="0" y="4" width="5" height="16" rx="1" fill="#f5a524"/>
<rect x="8" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.78"/>
<rect x="14" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.52"/>
<rect x="20" y="4" width="3" height="16" rx="1" fill="#f5a524" fill-opacity="0.28"/>
</svg>

After

Width:  |  Height:  |  Size: 481 B

View File

@ -0,0 +1,43 @@
---
// Footer override that preserves Starlight's default chrome
// (prev/next pagination, last-updated, edit link) and appends
// a Supported Systems "joint" badge below.
import Default from '@astrojs/starlight/components/Footer.astro';
---
<Default><slot /></Default>
<aside class="ifx-ss-badge" aria-label="Supported Systems">
<a class="ifx-ss-badge__link" href="https://supported.systems" rel="noopener">
<img
class="ifx-ss-badge__logo"
src="/supported-systems-logo.svg"
alt=""
width="60"
height="45"
loading="lazy"
/>
<div class="ifx-ss-badge__copy">
<h3 class="ifx-ss-badge__heading">A Supported Systems Joint</h3>
<p class="ifx-ss-badge__body">
<code>informix-driver</code> is built and maintained by
<span class="ifx-ss-badge__name">Supported Systems</span> &mdash; a
boutique software studio focused on thoughtful, user-first technology.
We take databases personally.
</p>
<span class="ifx-ss-badge__cta">
Visit supported.systems
<svg viewBox="0 0 16 16" width="14" height="14" aria-hidden="true">
<path
d="M4 8h7M8 5l3 3-3 3"
fill="none"
stroke="currentColor"
stroke-width="1.5"
stroke-linecap="round"
stroke-linejoin="round"
/>
</svg>
</span>
</div>
</a>
</aside>

View File

@ -0,0 +1,79 @@
---
// Custom hero for the homepage. Renders the wire-dump easter egg
// alongside the headline + CTA stack. Uses real captured SQLI bytes
// from docs/CAPTURES/01-connect-only.socat.log (truncated and curated).
---
<section class="ifx-hero">
<div class="ifx-hero__copy">
<span class="ifx-hero__eyebrow">Pure Python · No CSDK · No JVM · No libcrypt.so.1</span>
<h1 class="ifx-hero__title">
Talk to Informix without
<strong>linking against IBM's 92 MB tarball.</strong>
</h1>
<p class="ifx-hero__lede">
Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that.
So we read the protocol and wrote it ourselves: PEP 249, sync and async, pooled, TLS.
Within 10% of IBM's own C driver on bulk fetches, <strong>1.6× faster</strong> on
bulk inserts. No compile step. No <code>LD_LIBRARY_PATH</code> ritual. No
<code>libcrypt.so.1</code> from 2018.
</p>
<div class="ifx-hero__cta">
<a class="primary" href="/start/quickstart/">Get started →</a>
<a class="secondary" href="/start/vs-ifxpy/">Compared to IfxPy</a>
<a class="secondary" href="https://git.supported.systems/warehack.ing/informix-db">Source</a>
</div>
<div class="ifx-hero__install">pip install informix-driver</div>
</div>
<div class="ifx-hero__visual">
<div class="ifx-wiredump" aria-hidden="true">
<div class="ifx-wiredump__scroll" id="ifx-wire">
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--out">{'>'} OUT</span> 01 c3 01 3c 00 00 00 64 00 65 00 00 00 3d 00 06</span>
<span class="ifx-wiredump__line"> 49 45 45 45 4d 00 00 6c 73 71 6c 65 78 65 63 00 <span class="ifx-wiredump__byte--ascii">IEEEM..lsqlexec.</span></span>
<span class="ifx-wiredump__line"> 00 00 00 00 00 00 06 39 2e 32 38 30 00 00 0c 52 <span class="ifx-wiredump__byte--ascii">.......9.280...R</span></span>
<span class="ifx-wiredump__line"> 44 53 23 52 30 30 30 30 30 30 00 00 05 73 71 6c <span class="ifx-wiredump__byte--ascii">DS#R000000...sql</span></span>
<span class="ifx-wiredump__line"> 69 00 00 00 01 3c 00 00 00 00 00 00 00 00 00 01 <span class="ifx-wiredump__byte--ascii">i....{'<'}.........</span></span>
<span class="ifx-wiredump__line"> 00 09 69 6e 66 6f 72 6d 69 78 00 00 07 69 6e 34 <span class="ifx-wiredump__byte--ascii">..informix...in4</span></span>
<span class="ifx-wiredump__line"></span>
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--in">{'<'} IN </span> 01 14 02 3c 10 00 00 64 00 65 00 00 00 3d 00 06</span>
<span class="ifx-wiredump__line"> 49 45 45 45 49 00 00 6c 73 72 76 69 6e 66 78 00 <span class="ifx-wiredump__byte--ascii">IEEEI..lsrvinfx.</span></span>
<span class="ifx-wiredump__line"> 00 00 00 00 00 00 2f 49 42 4d 20 49 6e 66 6f 72 <span class="ifx-wiredump__byte--ascii">....../IBM Infor</span></span>
<span class="ifx-wiredump__line"> 6d 69 78 20 44 79 6e 61 6d 69 63 20 53 65 72 76 <span class="ifx-wiredump__byte--ascii">mix Dynamic Serv</span></span>
<span class="ifx-wiredump__line"> 65 72 20 56 65 72 73 69 6f 6e 20 31 35 2e 30 2e <span class="ifx-wiredump__byte--ascii">er Version 15.0.</span></span>
<span class="ifx-wiredump__line"></span>
<span class="ifx-wiredump__line"><span class="ifx-wiredump__direction ifx-wiredump__direction--out">{'>'} OUT</span> 00 02 00 00 00 00 00 49 73 65 6c 65 63 74 20 46 <span class="ifx-wiredump__byte--ascii">.......Iselect F</span></span>
<span class="ifx-wiredump__line"> 49 52 53 54 20 31 20 73 69 74 65 20 66 72 6f 6d <span class="ifx-wiredump__byte--ascii">IRST 1 site from</span></span>
<span class="ifx-wiredump__line"> 20 69 6e 66 6f 72 6d 69 78 2e 73 79 73 74 61 62 <span class="ifx-wiredump__byte--ascii"> informix.systab</span></span>
<span class="ifx-wiredump__line"> 6c 65 73 20 77 68 65 72 65 20 74 61 62 6e 61 6d <span class="ifx-wiredump__byte--ascii">les where tabnam</span></span>
<span class="ifx-wiredump__line"> 65 20 3d 20 27 20 47 4c 5f 43 4f 4c 4c 41 54 45 <span class="ifx-wiredump__byte--ascii">e = ' GL_COLLATE</span></span>
</div>
</div>
<div class="ifx-wiredump__caption">
live capture · 01-connect-only.socat · sqli/9088
</div>
</div>
</section>
<script>
// Reveal lines progressively with reduced-motion respect.
const reduce = window.matchMedia('(prefers-reduced-motion: reduce)').matches;
const root = document.getElementById('ifx-wire');
if (root) {
const lines = Array.from(root.querySelectorAll('.ifx-wiredump__line'));
if (reduce) {
lines.forEach((l) => l.classList.add('is-visible'));
} else {
let i = 0;
const reveal = () => {
if (i < lines.length) {
lines[i].classList.add('is-visible');
i++;
setTimeout(reveal, 110 + Math.random() * 60);
}
};
// Defer start until after first paint so it feels like a fresh capture.
requestAnimationFrame(() => setTimeout(reveal, 380));
}
}
</script>

View File

@ -0,0 +1,7 @@
import { defineCollection } from 'astro:content';
import { docsLoader } from '@astrojs/starlight/loaders';
import { docsSchema } from '@astrojs/starlight/schema';
export const collections = {
docs: defineCollection({ loader: docsLoader(), schema: docsSchema() }),
};

View File

@ -0,0 +1,81 @@
---
title: Architecture overview
description: How the layers stack, from socket through framing, codec, resultset, cursor, connection, and pool.
sidebar:
order: 2
---
The driver is six layers, each with a single responsibility, each testable in isolation.
```text
┌──────────────────────────────────────────────────┐
│ Connection / Pool │ ← public API
├──────────────────────────────────────────────────┤
│ Cursor │ ← PEP 249 surface
├──────────────────────────────────────────────────┤
│ ResultSet │ ← row iteration, prefetch
├──────────────────────────────────────────────────┤
│ Codec / Per-column │ ← decode SQL types → Python
│ readers │
├──────────────────────────────────────────────────┤
│ Protocol / PDU framing │ ← SQLI PDUs over the wire
├──────────────────────────────────────────────────┤
│ IfxSocket (buffered) │ ← raw bytes, recv() management
└──────────────────────────────────────────────────┘
```
## IfxSocket
The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies.
Everything above this layer is `bytes` and `bytearray` arithmetic, with no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser.
## Protocol / PDU framing
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean*, only how to read and write the byte shapes.
The PDU types and their fields were reverse-engineered from three sources:
1. The decompiled IBM JDBC driver (`com.informix.jdbc.IfxConnection` and the `IfxProtocol` class hierarchy).
2. Annotated `socat` captures of real client/server exchanges (`docs/CAPTURES/`).
3. Differential testing against IfxPy on identical data.
## Codec / Per-column readers
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types. See [SQL ↔ Python types](/reference/types/) for the full table.
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]`, with no per-column dispatch overhead.
Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code.
## ResultSet
`_resultset.py`. Holds the cursor's column descriptors, the pre-baked decoder list, and the in-flight prefetch state. Manages `fetch_one`, `fetch_many`, `fetch_all` semantics.
Most cursor calls land here: when you do `cur.fetchone()`, the cursor delegates to its ResultSet which reads the next `SQ_TUPLE` PDU, runs it through the per-column decoders, and returns a tuple.
## Cursor
`cursors.py`. The PEP 249 cursor surface. `execute()`, `executemany()`, `fetchone()`, `fetchmany()`, `fetchall()`, `description`, `rowcount`, scrollable cursor support.
This is where parameter binding lives: a SQL statement with `?` placeholders gets prepared via `SQ_PREPARE`, the driver introspects the parameter shape via `SQ_DESCRIBE`, and bound values get encoded according to the parameter types.
## Connection / Pool
`connections.py` and `pool.py`. The connection owns the IfxSocket, manages transaction state, and is the parent of all cursors. The pool is a wrapper around N connections with the usual acquire/release/timeout semantics.
`aio.py` mirrors all of the above with `async def` versions, implemented via thread-pool wrapping (see [Async strategy →](/explain/async-strategy/)).
## Why this layering
Each boundary is testable in isolation:
- `IfxSocket` tests can use a `socket.socketpair()` and assert on byte streams.
- Protocol tests parse known-good captures from `docs/CAPTURES/` and verify the typed PDUs come out correctly.
- Codec tests pass synthetic byte payloads to per-column readers and assert the Python output.
- ResultSet tests can use a fake protocol that emits canned PDU sequences.
- Cursor tests use a fake ResultSet.
When a regression appears, the layered structure narrows the search: a wire-format test failing means it's the protocol layer; a row-tuple test failing with a corrupt-bytes input means the codec; a `fetchall` test failing means the ResultSet's iteration logic. The 300+ test suite leans heavily on this isolation.

View File

@ -0,0 +1,61 @@
---
title: Async strategy
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async, and what that costs.
sidebar:
order: 4
---
import { Aside } from '@astrojs/starlight/components';
`informix-driver`'s async API (`from informix_db import aio`) is implemented by wrapping the sync core in a thread pool. Every `await cur.execute(...)` schedules the underlying sync `execute()` on the pool's executor.
This is a deliberate architectural choice from Phase 16. Here's the reasoning.
## What we considered
Three options for adding async support to a sync database driver:
1. **Full async I/O refactor.** Rewrite the protocol layer on top of `asyncio.Protocol` or `asyncio.StreamReader`. The codec, framing, and connection state all become coroutines. ~2000 lines of code, full test rewrite.
2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes.
3. **Dual implementations.** Maintain two parallel code paths, one sync and one async. Most code duplicated. Worst of both worlds.
We picked option 2.
## Why option 2 was the right call
For typical database workloads, meaning request-scoped connections that mostly wait on I/O, the practical difference between option 1 and option 2 is small:
- **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~510 µs of executor overhead.
- **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10100 concurrent FastAPI requests on a 4-core box, this difference is small.
- **Code complexity**: option 1 is dramatically harder to write and test. The protocol layer becomes asynchronous everywhere; cancellation paths multiply; the shape of "what does a partial PDU read look like" becomes a state machine instead of a `while not done: read_more()`.
For a driver that needs to be production-ready in finite engineering time, option 2 was the right call.
## What it costs
The honest costs:
- **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process.
- **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor.
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe, in that cancelled workers cannot leak onto recycled pool connections, but the underlying syscall does still complete.
## What it doesn't cost
- **Cancellation safety.** This was the original concern. Phase 27's per-connection wire lock + worker reaping makes async cancellation cancellation-safe in the same sense `asyncpg` is.
- **FastAPI integration.** The `aio.Pool` is a drop-in replacement for any "async database pool" pattern. `Depends(get_conn)` works exactly as you'd expect.
- **Async generator support.** `async for row in cur` works. The fetch is per-row chunked through the executor; the iteration shape is async-native.
## When option 1 might still be worth it
The two scenarios where a full async I/O implementation would matter:
1. **Very high concurrency on a single process** (1000+ in-flight queries). Thread context-switching cost becomes measurable. We haven't hit this in practice.
2. **Sub-millisecond query latencies on a unloaded server.** The 510 µs executor overhead is a meaningful percentage. For typical Informix workloads where round-trip is ~80 µs+, it isn't.
If either becomes a real production concern, the layered architecture lets us swap in a fully-async lower half without changing the upper half. The cursor / connection / pool API doesn't care how the bytes get to and from the server. That's the option-2 win we explicitly preserved.
<Aside type="note">
This is a Phase 16 decision. The pivot from "rewrite as async-native" to "wrap the sync core" is documented in [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md). Three years from now, if it turns out we should have gone with option 1, we have a clear path.
</Aside>

View File

@ -0,0 +1,147 @@
---
title: The buffered reader
description: How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1× IfxPy by moving the recv() buffer one level down the object graph.
sidebar:
order: 3
---
import { Aside } from '@astrojs/starlight/components';
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39, a connection-scoped buffered reader, closed it from 2.4× to ~1.051.15× in about thirty minutes of code plus ten minutes of architectural debugging.
This page is about both the technical change and the failure mode that hid the win for two phases.
## The lever we couldn't see
After Phase 38, profiling a 100,000-row fetch showed:
| Category | Self time | % of wall clock |
|---|---:|---:|
| I/O machinery | 555 ms | 66% |
| Codec | 205 ms | 24% |
| Other | ~80 ms | 10% |
The headline "I/O dominated" was true. The interesting half is the breakdown of that 555 ms:
- Actual `recv()` syscalls: ~153 ms
- Python wrapper overhead: ~400 ms
That ~400 ms was our own buffer abstraction: a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
The kernel was doing maybe 2530 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves.
## The architecture pattern
Both `asyncpg` (in `buffer.pyx`) and `psycopg3` (in `pq.PGconn`) put a single growing read buffer on the protocol/connection object. The parser indexes into it via `struct.unpack_from(buf, offset)` rather than slicing copies. Refills happen via one large `recv(64K)` rather than many small `recv()`s for individual fields.
Phase 39 ports that pattern to `informix-driver`. The state machine:
```text
┌───────────────────────────────┐
│ IfxSocket │
│ ─ socket: socket.socket │
│ ─ buf: bytearray (growable) │
│ ─ offset: int (read cursor) │
│ │
│ recv(64K) when buf exhausted │
└───────────────────────────────┘
│ reads via read_exact(n)
┌───────────────────────────────┐
│ SocketReader (per-PDU) │
│ ─ short-lived view │
│ ─ no buffer of its own │
└───────────────────────────────┘
```
The reader is a parser-view. The buffer outlives the reader. When the parser asks for `read_short()`, the reader calls `socket.read_exact(2)`, which slices two bytes out of the bytearray at `offset` and advances. If the bytearray runs out, `socket.recv(64K)` refills it.
Result: **one `recv()` per ~64 KB of incoming data**, not per field.
## The architectural mistake the first pass got wrong
The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first.
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken`, meaning the process was blocked in `recv()` waiting for bytes that weren't coming.
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes, 200 for response 1 and 400 for response 2, it consumed bytes belonging to response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket`, the connection-scoped wrapper. Readers are short-lived parser-views, and the buffer outlives them.
```python
# WRONG (first pass): buffer scoped to reader
class BufferedSocketReader:
def __init__(self, sock):
self.sock = sock
self.buf = bytearray() # ← dies with the reader
self.offset = 0
# RIGHT (Phase 39): buffer scoped to connection
class IfxSocket:
def __init__(self, sock):
self.sock = sock
self._read_buf = bytearray() # ← outlives all readers
self._read_offset = 0
def read_exact(self, n):
if self._read_offset + n > len(self._read_buf):
self._refill()
...
```
asyncpg and psycopg3 both place the buffer on the protocol/connection object. The architectural template was sitting in front of me before I started; I built the wrong shape anyway because "buffered reader" implies the buffer is *on* the reader.
It is not. The reader is a view. The buffer is state.
## The numbers
A/B-measured against the same Docker container, warmed cache, only the env flag differing:
| Workload | Phase 38 | Phase 39 | Δ |
|---|---:|---:|---:|
| `select_scaling_1000` | 2.901 ms | 1.716 ms | **41%** |
| `select_scaling_10000` | 24.317 ms | 16.084 ms | **34%** |
| `select_scaling_100000` | 250.363 ms | 168.982 ms | **32%** |
Re-running the IfxPy comparison after Phase 39:
| Workload | IfxPy 2.0.7 (C) | informix-driver Phase 39 | Ratio |
|---|---:|---:|---:|
| `select_scaling_1000` | 1.637 ms | 1.716 ms | **1.05×** |
| `select_scaling_10000` | 15.07 ms | 16.08 ms | **1.07×** |
| `select_scaling_100000` | 147.4 ms | 169.0 ms | **1.15×** |
The 2.4× steady-state gap that existed before Phase 37 is now within 515% of the C driver, and the lower bound may already be within IfxPy's own measurement noise (its IQR on the 100k workload is 21%; ours is 0.2%).
## What the feature flag does
The buffered reader ships **enabled by default** in version 2026.05.05.12. To opt out (debugging, regressing a workload, A/B-measuring your own code):
```bash
IFX_BUFFERED_READER=0 python my_app.py
```
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime, so close and reopen the pool to flip behavior.
<Aside type="note">
The flag exists to make A/B measurement easy. There's no expected reason to disable it in production. If you hit a workload where the buffered reader is slower, that's a bug and we'd like to know.
</Aside>
## What we learned
The general pattern: **what's visible gets optimization attention; what's invisible gets written off as irreducible**.
The codec is visible: there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads (`read_short`, `read_int`, `read_exact(payload_size)`) are one-liners. What could be slow about that?
It was carrying ~30% of total wall time. Two phases of changelogs implicitly blamed "the protocol" for the remaining gap. The actual culprit was a few lines of `bytes.join` in a wrapper from Phase 1 that nobody had revisited.
The lesson is small and easy to state: a profile turns vibes into an attack surface. Write the closing paragraph after you've measured, not before.
## Read more
- **[Architecture overview →](/explain/architecture/)**: where the buffered reader sits in the layer stack.
- **[Phase log →](/explain/phase-log/)**: the full progression from Phase 1 through Phase 39+.
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)**: Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.

View File

@ -0,0 +1,150 @@
---
title: The phase log
description: Phase-by-phase narrative of how informix-driver got built, with notable architectural decisions called out.
sidebar:
order: 6
---
The driver was built across 39+ phases, each with a focused scope and a decision log. This page is a high-level index; the gory details (with rationale, alternatives considered, and rollback notes) live in [`docs/DECISION_LOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md).
## Foundation (Phases 110)
| Phase | Title | Outcome |
|---|---|---|
| 1 | Socket + minimal SQ_INFO | First handshake against the dev container |
| 24 | Login, DBOPEN, error decoding | Can connect and select a database |
| 5 | Statement execution | `SELECT 1` works |
| 6 | Parameter binding | `?`-placeholders, basic types |
| 7 | Logged-DB transactions | Discovered Informix needs explicit `SQ_BEGIN` per tx in non-ANSI mode |
| 8 | BYTE / TEXT (legacy in-row blobs) | Needs blobspace |
| 9 | Scrollable cursors | `SQ_SCROLL` PDU, position semantics |
| 10/11 | Smart-LOB read & write | **Architectural pivot** to `SQ_FILE` intercept; ~3× smaller than projected |
## Hardening (Phases 1220)
| Phase | Title | Outcome |
|---|---|---|
| 12 | Type system overhaul | Per-column readers (predecessor to Phase 37) |
| 13 | DECIMAL / MONEY exact precision | `decimal.Decimal` round-trip |
| 14 | DATETIME range typing | Returns `date` / `datetime` / `time` per field range |
| 15 | INTERVAL types | Custom `IntervalYM`, `timedelta` for D-to-F |
| 16 | Async API | **Pivot** to thread-pool wrapping (~250 lines) instead of full async refactor (~2000 lines) |
| 17 | Connection pool (sync) | min/max sizing, acquire timeout, max idle |
| 18 | Connection pool (async) | Mirror of sync API on `aio.Pool` |
| 19 | TLS support | Bring-your-own-context, `tls=True` for dev |
| 20 | Locale / Unicode | `client_locale`, full mapping in [`Connection.encoding`](/reference/types/) |
## Production review (Phases 2130)
| Phase | Title | Outcome |
|---|---|---|
| 21 | Type-checking pass | `py.typed`, full mypy/pyright coverage |
| 22 | Error code mapping | SQLCODE → exception per [reference](/reference/exceptions/) |
| 23 | Health checks | Pool validates idle connections before return |
| 24 | Statement caching | Per-connection prepared-statement cache |
| 25 | Fast-path call (`SQ_FPROUTINE`) | Direct UDF/SPL invocation, bypassing PREPARE |
| 26 | **CRITICAL** | Pool returned connections with open transactions (fixed) |
| 27 | **CRITICAL** | Per-connection wire lock + async cancellation safety |
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync (fixed) |
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise (fixed) |
| 30 | Hardening pass | 5 medium-severity audit findings, all closed |
After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver is production-ready.
## Performance (Phases 3139)
| Phase | Title | Result |
|---|---|---|
| 31 | Statement cache LRU tuning | Better hit rate on repeated queries |
| 32 | Cursor lifecycle optimization | Fewer round-trips on small queries |
| 33 | **Pipelined `executemany`** | **1.6× faster than IfxPy on bulk inserts** |
| 34 | LRU caches for type lookup | Removed dispatch overhead on hot paths |
| 35 | Memory profile pass | Identified 100k-row baseline |
| 36 | `IfxPy` comparison harness | Established the 2.4× bulk-fetch gap |
| 37 | **Per-column reader strategy** | 10% on bulk SELECT, ratio → 2.10× |
| 38 | **`exec()`-based row-decoder codegen** | Further 12%, ratio → 2.04× |
| 39 | **Connection-scoped buffered reader** | **32% on bulk SELECT, ratio → 1.051.15×** |
The Phase 3739 trajectory is documented in detail at [The buffered reader →](/explain/buffered-reader/), including the architectural mistake the first pass got wrong.
## Field reports and systematic review (2026.082026.09)
Phase 30's audit found nothing left. Then real schemas arrived, and the audit's clean bill of health turned out to be a statement about the questions it had asked rather than about the driver.
A field report from a user running Informix 12 surfaced three type-framing bugs in a single afternoon: `INT8` / `SERIAL8` never decoded at all, `NCHAR` losing its first character, and `BOOLEAN` corrupting every column after it. Fuzzing the type matrix found more, and each one had the same shape as the last.
`2026.09.02` went back over the driver looking for that shape rather than for new symptoms, and found eleven bugs.
| Area | What was wrong |
|---|---|
| Row framing | Nothing ever checked that a row consumed its own payload. Fourteen framing bugs had reached users, every one detectable for free |
| Smart LOBs | Read as a flat 72-byte field when they occupy 149, shifting every column after a non-final `BLOB` |
| Composites | A NULL collection skipped its length field, the same bug as NULL `LVARCHAR` twelve lines away |
| Transactions | `rollback()` after a SQL `BEGIN WORK` sent nothing and reported success |
| Classification | Comments, CTEs and parenthesized selects were run as DML and failed with `-260` |
| Scrollable cursors | A second statement on the connection got `-285` and destroyed the cursor too |
| Statement release | Two of eight exits leaked a failed statement, bricking the connection |
| Cursor finalizers | Cleanup could land inside another statement, and the lock probe was blind to its own thread |
| Async | The whole layer shared the process-wide thread pool, so cancellations starved it |
| Socket | Two readers on one stream agreed only by accident of how the server replies |
The pattern behind almost all of them: a hazard that was understood and guarded at the site where it was first observed, rather than at the abstraction that owned it. The guard then never travelled to its siblings. Four separate hand-written copies of the same UDT envelope, three of them wrong. Six hand-rolled copies of the same statement-release cleanup, two missing entirely.
Two of the fixes came from asking the server instead of guessing. `statement_type` and `statement_id` had both been parsed out of every DESCRIBE response into a metadata dict that nothing read, while the code that needed them inferred the answer from the first word of the SQL and got it wrong five ways.
The cheapest fix was also the most valuable. Asserting that a row decoder lands exactly on the end of its payload costs one integer comparison, and it would have caught all fourteen framing bugs at the byte where each happened rather than three releases later in somebody's result set.
Test count went from 241 to 457 across three server versions over this stretch.
### 2026.09.03
Two more, one of each kind.
The `:N` placeholder rewriter was a bare regular expression, so it
rewrote the inside of string literals: `'http://host:8080/x'` was stored
as `'http://host?/x'`. Same family as the statement-classification bug
above, and the last place the driver inferred meaning from SQL text
instead of handling it. Replaced with a scanner using Informix's own
lexical rules, measured rather than assumed. One of those rules would
have been got wrong from habit: a backslash escapes nothing, so `'a\'b'`
is an unterminated string and draws `-282`. A scanner written to
Postgres reflexes desyncs there and corrupts everything after it.
`row_factory=Row` adds column-name and attribute access, on request from
a user coming from `pyodbc`. Opt-in, because it costs about 9% on bulk
fetch and that is not a trade to make for someone who never reads a
column by name.
Testing the second one found the pattern again, this time in code
written an hour earlier. The names a column could shadow were
hand-listed as `count` and `index`, the two `tuple` methods; then `keys`,
`_asdict` and `_fields` were added to the same class and the list was
never revisited. A column called `keys` returned a bound method, in
silence. The fix was to compute the set from the class rather than
lengthen the list, and to parametrize the test over that computed set so
it grows with the class.
The interesting part is where testing found it. A differential test
across twenty columns covering every awkward type, populated and fully
NULL, found nothing. The bug was in six lines of name bookkeeping that
looked obviously correct, and had been correct when written. Anything
that must stay in sync with a class by hand eventually will not.
## Notable architectural pivots
The decision log calls out four moments where the obvious choice would have been wrong:
1. **Phase 10/11**: abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
2. **Phase 16**: thread-pool async instead of a full async refactor. ~88% less code, same FastAPI surface.
3. **Phase 27**: adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
4. **Phase 39**: buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
## What's next
The roadmap (loose, not committed):
- **Phase 40+ (codec)**: Numpy-backed bulk decode for homogeneous columns. ~5× speedup target on analytical workloads.
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python", so it would sit behind a build flag.
- **Phase 5x (API)**: Native `callproc` with named parameters, IBM-specific scrollable cursor extensions for full IfxPy parity.
The phase log is updated as work lands. The repo's [`CHANGELOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md) is the source of truth for shipped changes.

View File

@ -0,0 +1,63 @@
---
title: Pure-Python tradeoffs
description: What pure-Python costs, what it pays for, and where the actual performance ceiling sits.
sidebar:
order: 5
---
The premise of this driver is *zero native code in the call stack*. No CSDK, no JDBC, no C extension we maintain ourselves. Just `socket`, `struct`, `decimal`, and the standard library.
This page is about what that costs and what it pays for.
## What pure-Python costs
The honest accounting:
| Cost | Magnitude | Mitigation |
|---|---|---|
| Per-row decode overhead | ~2.0 µs/row vs IfxPy's ~1.1 µs/row | Phases 3738 codec inlining brought us from 4 µs to 2 µs. |
| Per-PDU parser overhead | ~510 µs vs C's ~1 µs | Phase 39 buffered reader removed the worst of it (the read-side wrapper cost). |
| GIL contention on multi-threaded decode | Threads serialize through codec hot loops | Pool gives one connection per thread; codec releases GIL during I/O. |
| Memory per connection | ~50500 KB (Phase 39 buffer) | Pool keeps it bounded; freed on connection close. |
The order-of-magnitude intuition: pure-Python is ~2× slower than C-bound for **codec-bound workloads** (large analytical fetches), and **competitive or faster** for I/O-bound workloads (transactional, bulk-insert, FastAPI request-response).
## What pure-Python pays for
The benefits are mostly deployment, not performance:
- **50 KB wheel**: installable in a slim Docker image without a build toolchain.
- **No `libcrypt.so.1`**: works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
- **Python 3.103.14**: no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
- **Type annotations everywhere**: `py.typed` flag, full coverage in mypy / pyright.
- **Auditable codepaths**: every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
- **Async without `run_in_executor`**: native `async def` API, FastAPI-compatible.
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1`: those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
## Where the ceiling sits
For codec-bound workloads, the ceiling we've hit is around 2 µs/row for tabular data. The breakdown:
- `struct.unpack_from(format, buf, offset)` per field: ~80 ns
- `bytes``str` decoding (varchar): ~150 ns
- Tuple construction: ~100 ns
- Cursor / ResultSet bookkeeping: ~50 ns
Five fields × ~250 ns/field + ~250 ns overhead = ~1.5 µs. We're at ~2.0 µs which means ~30% overhead remains. That's the gap between "we've inlined everything that's reasonable" and "the C version still wins."
Strategies for closing further:
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim, because there'd be a build step.
- **Bytecode optimization via `exec()`-codegen** (the Phase 38 approach). Marginal further wins; we've already extracted most of what's available.
- **Numpy-backed bulk decode** for homogeneous columns. Promising for analytical workloads. ~5× speedup possible for `SELECT col FROM huge_table` over the current per-row approach. Probably Phase 41+.
For I/O-bound workloads we're already at the ceiling. The buffered reader closed the I/O gap; further wins are at the kernel level (e.g. `recvmsg` for vectored reads), which is moot since the kernel already isn't the bottleneck.
## The honest summary
Pure-Python costs us ~515% on bulk-fetch workloads and zero (or favorable) on everything else. The deployment, async, and modern-Python wins are large and don't depend on workload.
If the codec gap matters for your case, meaning analytical reporting against a wide table or pulling millions of rows in a single SELECT, IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
The driver chose the goal, *first pure-socket Informix driver in any language*, over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.

View File

@ -0,0 +1,84 @@
---
title: The SQLI wire protocol
description: A short tour of Informix's SQLI protocol, covering PDU framing, the handshake, statement execution, and fetch.
sidebar:
order: 1
---
import { Aside } from '@astrojs/starlight/components';
SQLI is Informix's wire protocol, the same one IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
This page is a short tour. The byte-level reference (with hex annotations) lives in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) in the repo.
## PDU framing
Every PDU starts with a 1-byte tag and (mostly) ends with a 2-byte EOT marker. Common tags:
| Tag | Hex | Direction | Purpose |
|---|---|---|---|
| `SQ_INFO` | `0x01` | →S | Initial capability/identity exchange |
| `SQ_VERSION` | `0x14` | S→ | Server version response |
| `SQ_PASSWD` | `0x18` | →S | Authentication |
| `SQ_DBOPEN` | `0x24` | →S | Open database |
| `SQ_PREPARE` | `0x02` | →S | Prepare statement |
| `SQ_DESCRIBE` | `0x08` | →S | Describe column structure |
| `SQ_OPEN` | `0x06` | →S | Open cursor |
| `SQ_FETCH` | `0x04` | →S | Fetch rows |
| `SQ_TUPLE` | `0x09` | S→ | One row of data |
| `SQ_ID` | `0x37` | S→ | SQLCODE / status code |
| `SQ_FILE` | `0x62` | both | Smart-LOB transfer |
| `SQ_EOT` | `0x0c` | both | End-of-transmission |
The trailing `00 0c` (length=0, tag=0x0c) marks the end of every multi-PDU response.
## A connect, in bytes
Annotated output from `docs/CAPTURES/01-connect-only.socat.log`:
```text
> OUT 01 c3 01 3c 00 00 00 64 00 65 00 00 00 3d 00 06 ; SQ_INFO
49 45 45 45 4d 00 00 6c 73 71 6c 65 78 65 63 00 ; "IEEEM..lsqlexec"
00 00 00 00 00 00 06 39 2e 32 38 30 00 00 0c 52 ; ".......9.280...R"
44 53 23 52 30 30 30 30 30 30 00 00 05 73 71 6c ; "DS#R000000...sql"
69 00 00 00 01 3c 00 00 00 00 00 00 00 00 00 01 ; "i....<.........."
...
< IN 01 14 02 3c 10 00 00 64 00 65 00 00 00 3d 00 06 ; SQ_VERSION
49 45 45 45 49 00 00 6c 73 72 76 69 6e 66 78 00 ; "IEEEI..lsrvinfx."
00 00 00 00 00 00 2f 49 42 4d 20 49 6e 66 6f 72 ; "....../IBM Infor"
6d 69 78 20 44 79 6e 61 6d 69 63 20 53 65 72 76 ; "mix Dynamic Serv"
65 72 20 56 65 72 73 69 6f 6e 20 31 35 2e 30 2e ; "er Version 15.0."
```
The payload of `SQ_INFO` is a sequence of length-prefixed strings: byte-order marker (`IEEEM` = big-endian), client app name (`sqlexec`), client version (`9.280`), build ID, protocol token (`sqli`), feature flags. The server's `SQ_VERSION` response mirrors this with the server's own identification.
## Statement execution
The full lifecycle for `SELECT id FROM users WHERE id = ?`:
```text
→ SQ_PREPARE "SELECT id FROM users WHERE id = ?"
← SQ_ID (statement ID, parameter shape, ...)
→ SQ_DESCRIBE
← SQ_DESC (column descriptors: "id" SMALLINT)
→ SQ_OPEN (parameter values: 42)
← SQ_ID (cursor ID)
→ SQ_FETCH
← SQ_TUPLE (id=42)
← SQ_TUPLE (or SQ_DONE)
→ SQ_CLOSE (release cursor)
← SQ_ID
```
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts. See [Bulk inserts](/how-to/executemany/).
## Smart-LOB transfer
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side, with no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
This was the architectural pivot in [Phase 10/11](/explain/phase-log/) that made smart-LOBs work end-to-end in pure Python. Reading and writing GB-sized BLOBs goes through the same socket as any other query.
<Aside type="note">
The protocol has many more PDU types than this page covers, mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
</Aside>

View File

@ -0,0 +1,93 @@
---
title: Async with FastAPI
description: Wire the async pool into a FastAPI app with proper lifecycle management.
sidebar:
order: 3
---
import { Aside } from '@astrojs/starlight/components';
`informix-driver` has a native async API. Use it from FastAPI by creating the pool in the app's lifespan and dependency-injecting connections per request.
## App skeleton
```python
from contextlib import asynccontextmanager
from fastapi import FastAPI, Depends, HTTPException
from informix_db import aio
@asynccontextmanager
async def lifespan(app: FastAPI):
app.state.pool = await aio.create_pool(
host="db.example.com",
user="informix", password="...",
database="mydb", server="informix",
min_size=2, max_size=20,
)
yield
await app.state.pool.close()
app = FastAPI(lifespan=lifespan)
async def get_conn():
async with app.state.pool.connection() as conn:
yield conn
@app.get("/users/{user_id}")
async def get_user(user_id: int, conn = Depends(get_conn)):
cur = await conn.cursor()
await cur.execute(
"SELECT id, name, email FROM users WHERE id = ?",
(user_id,),
)
row = await cur.fetchone()
if row is None:
raise HTTPException(404, "user not found")
return {"id": row[0], "name": row[1], "email": row[2]}
```
## Why this shape
- **Lifespan-scoped pool**: the pool lives for the lifetime of the app, login handshake amortized across all requests.
- **Per-request connection via `Depends`**: each request gets its own connection from the pool. The async generator pattern (`yield conn`) means the connection returns to the pool when the request finishes, including on exception.
- **No `run_in_executor`**: every call is `await`able natively. No event-loop blocking, no thread-pool tuning.
## Cancellation
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe: the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean, with transactions rolled back. You don't need to wrap anything in `try/finally`.
<Aside type="note">
This is a Phase 27 invariant: async cancellation cannot leak running workers onto recycled connections. The earlier behavior was a `High` audit finding; the fix is a CI tripwire test that's been green every commit since.
</Aside>
## Connection-level transactions
For request-scoped transactions, commit on the way out and roll back on any exception:
```python
@app.post("/orders")
async def create_order(order: OrderIn, conn = Depends(get_conn)):
cur = await conn.cursor()
try:
await cur.execute(
"INSERT INTO orders VALUES (?, ?, ?)",
(order.id, order.customer_id, order.total),
)
await cur.execute(
"UPDATE inventory SET qty = qty - ? WHERE sku = ?",
(order.qty, order.sku),
)
await conn.commit()
except Exception:
await conn.rollback()
raise
return {"ok": True}
```
`HTTPException` is an `Exception`, so raising one inside the block rolls back before FastAPI turns it into a response.
<Aside type="note">
There is no `conn.transaction()` context manager on either the sync or async connection, though an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
</Aside>

View File

@ -0,0 +1,48 @@
---
title: Optimize bulk SELECT
description: How the buffered reader works in practice, when it's on, when it isn't, and how to A/B-measure your workload.
sidebar:
order: 5
---
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything, since the bulk-fetch gap against IfxPy is now ~515% rather than ~140%.
For the architectural rationale, see [The buffered reader →](/explain/buffered-reader/).
## Disabling the buffered reader
```bash
IFX_BUFFERED_READER=0 python my_app.py
```
The flag is read once at connection construction. To flip behavior on existing connections, close and reopen the pool.
There's no expected reason to disable it in production. The flag exists so you can A/B-measure your own workload and so we can debug regressions if they appear.
## A/B-measuring your workload
```bash
# Baseline: no buffered reader
IFX_BUFFERED_READER=0 python -m mybench
# With buffered reader
IFX_BUFFERED_READER=1 python -m mybench
```
For typical bulk-SELECT workloads expect a 3040% wall-time reduction. For workloads dominated by single-row queries the impact is small (small queries are RTT-bound, not framing-bound).
## When the speedup is largest
Workloads where every column read makes ~45 small `recv()` calls, meaning tabular data, narrow rows, and large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
| Workload shape | Speedup |
|---|---:|
| Wide row, single fetch (1 row × 100 cols) | minimal |
| Narrow row, large fetch (100k rows × 5 cols) | 3040% |
| `executemany` response drain (1k inserts) | 2530% |
## Memory profile
The buffer is per-connection, sized to grow up to the largest single PDU it sees. Typical steady-state: 64 KB to a few hundred KB per connection. The buffer is freed when the connection closes; for long-lived pool connections it's amortized.
If you're running 10,000 connections at idle, the buffer cost is ~12 GB across the fleet. For typical pool sizes (1050 connections) it's ~110 MB total.

View File

@ -0,0 +1,93 @@
---
title: Run the dev container
description: IBM Informix Developer Edition in Docker, covering first-time setup, sbspace for smart-LOBs, and common troubleshooting.
sidebar:
order: 8
---
import { Aside } from '@astrojs/starlight/components';
The IBM Informix Developer Edition Docker image is the recommended dev / integration-test target. It's the same image our CI runs against.
## First-time setup
```bash
docker run -d --name informix-dev \
-e LICENSE=accept \
-p 9088:9088 \
-p 9089:9089 \
--privileged \
icr.io/informix/informix-developer-database:15.0.1.0.3DE
```
- **`9088`**: clear-text SQLI listener
- **`9089`**: TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
- **`--privileged`**: required for the dev image's shared-memory tuning
The image takes ~90 seconds to initialize. Watch for `oninit running`:
```bash
docker logs -f informix-dev
```
## Default credentials
| Field | Value |
|---|---|
| `user` | `informix` |
| `password` | `in4mix` |
| `database` | `sysmaster` (always exists) |
| `server` | `informix` |
For application use create your own database:
```bash
docker exec -it informix-dev bash -c '
echo "CREATE DATABASE app_db WITH LOG;" | dbaccess sysmaster
'
```
## Setup for smart-LOB tests
Smart-LOBs (BLOB / CLOB) require additional one-time setup:
```bash
# Inside the container
docker exec -it informix-dev su - informix -c '
onspaces -c -S sbspace1 -p $INFORMIXDIR/sbspace1 -o 0 -s 100000
# Edit $ONCONFIG to set SBSPACENAME sbspace1, then:
onmode -ky
oninit -y
# Take a level-0 archive so the sbspace is usable
ontape -s -L 0
'
```
After that, BLOBs and CLOBs work end-to-end. See [`docs/DECISION_LOG.md` §10](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md) for the gory details.
## Running the integration tests
```bash
make ifx-up # starts the container if not already running
make test-integration # runs the 231 integration tests
```
Or directly:
```bash
pytest -m integration
```
## Troubleshooting
**"Connection refused" on 9088**: the image is still initializing. Wait for `oninit running` in the logs.
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing, since it always exists.
**`-908` (system error / shared memory)**: the container needs `--privileged`. Restart with that flag.
<Aside type="tip">
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart, and the database survives container shutdown.
</Aside>

View File

@ -0,0 +1,107 @@
---
title: Bulk inserts (executemany)
description: How to bulk-load with executemany, including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
sidebar:
order: 4
---
import { Aside } from '@astrojs/starlight/components';
`executemany()` is the right tool for bulk inserts and updates. With `informix-driver`'s pipelined implementation it's **1.6× faster than IfxPy** at 10k+ rows.
## The basic shape
```python
rows = [(1, "alice"), (2, "bob"), (3, "carol"), ...] # 10_000 tuples
with conn: # opens a transaction
cur = conn.cursor()
cur.executemany(
"INSERT INTO users (id, name) VALUES (?, ?)",
rows,
) # commits on normal exit
```
That inserts 10,000 rows in ~161 ms against a loopback Informix container.
## The 53× gotcha
<Aside type="caution">
**`executemany(...)` under `autocommit=True` is 53× slower than the same call inside an explicit transaction.**
The server flushes the transaction log to disk per row in autocommit mode. With 10,000 single-row autocommit inserts that's 10,000 log flushes. Inside one transaction it's one flush at commit.
</Aside>
```python
# SLOW: 8.5 seconds for 10k rows
conn = informix_db.connect(..., autocommit=True)
cur = conn.cursor()
cur.executemany("INSERT ...", rows)
# FAST: 161 ms for 10k rows
conn = informix_db.connect(..., autocommit=False) # default
with conn:
cur = conn.cursor()
cur.executemany("INSERT ...", rows)
```
The default is `autocommit=False`, so this only catches you if you've explicitly opted into autocommit.
## Why it's faster than IfxPy
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row, so for 10,000 rows on an 80 µs RTT it adds up to 800 ms of just waiting for ACKs.
Phase 33 changed our `executemany` to **pipeline** the BIND+EXECUTE PDUs:
1. Send all N BIND PDUs back-to-back without reading responses
2. Send all N EXECUTE PDUs back-to-back
3. Drain N response sets at the end
The kernel buffers the outbound bytes; the server processes the BIND/EXECUTE pipeline as fast as it can; we read all responses at the end. One RTT for the whole batch instead of N.
## Chunking large batches
For very large batches (millions of rows), break into chunks to bound memory:
```python
def chunks(it, n):
buf = []
for x in it:
buf.append(x)
if len(buf) >= n:
yield buf
buf = []
if buf:
yield buf
with conn:
cur = conn.cursor()
for batch in chunks(huge_iterator, 10_000):
cur.executemany("INSERT INTO logs ...", batch)
# one transaction, many batched executemany calls, one commit at the end
```
10,000 rows per chunk is a reasonable default; the per-chunk Python memory cost is `~ N × bytes_per_row`. For 10k tuples of 5 small fields that's a few MB.
## Returning generated keys
Informix's `SERIAL` columns are server-assigned, and `executemany` doesn't return per-row IDs. To get an ID back, insert one row at a time and ask the server for the value it assigned:
```python
cur.execute("INSERT INTO orders (customer_id, total) VALUES (?, ?)", (7, 42.0))
cur.execute("SELECT DBINFO('sqlca.sqlerrd1') FROM systables WHERE tabid = 1")
new_id = cur.fetchone()[0]
```
`sqlca.sqlerrd1` is where Informix records the serial value from the most recent INSERT on this connection, so read it before running any other statement.
There is no `cursor.lastrowid`, though an earlier version of this page said there was.
For batch inserts that need the IDs, the idiomatic pattern is:
```sql
INSERT INTO orders (id, customer_id, total)
SELECT MAX(id) + ROWNUM, ?, ? FROM orders
```
…or pre-generate IDs from a sequence in your app code. Or accept `IfxPy`'s same limitation: `IfxPy.executemany` doesn't return per-row generated keys either.

View File

@ -0,0 +1,98 @@
---
title: Migrate from IfxPy
description: API differences between IfxPy and informix-driver, what's the same, what's not, and how to migrate incrementally.
sidebar:
order: 7
---
If you have working IfxPy code, migration is mostly mechanical. Both drivers are PEP 249 with similar shapes; the differences are in connection construction, a few cursor extensions, and async support.
## Connection construction
```python
# IfxPy
import IfxPy
conn_str = (
"DATABASE=mydb;HOSTNAME=db.example.com;PORT=9088;"
"PROTOCOL=onsoctcp;UID=informix;PWD=...;SERVICE=informix"
)
conn = IfxPy.connect(conn_str, "", "")
# informix-driver
import informix_db
conn = informix_db.connect(
host="db.example.com", port=9088,
user="informix", password="...",
database="mydb", server="informix",
)
```
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported, deliberately: they're a security and parsing footgun.
## The DB-API surface is the same
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, and `close()` all behave per PEP 249.
The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `DatabaseError`, `DataError`, `OperationalError`, `IntegrityError`, `InternalError`, `ProgrammingError`, `NotSupportedError`.
## What IfxPy has that we don't (yet)
- **Named-parameter `callproc`**. We have `fast_path_call` for direct UDF/SPL invocation but the API shape differs.
- **IBM-specific scrollable cursor extensions**. We support PEP 249 scrollable cursors (`scroll(value, mode)`) but not IfxPy's `last`/`prior`/`relative` shortcuts.
- **`cursor.set_chunk_size`**. We tune fetch behavior via the buffered reader; no per-cursor knob.
## What we have that IfxPy doesn't
- **Async API** (`from informix_db import aio`)
- **Connection pool** (`informix_db.create_pool` / `aio.create_pool`)
- **Type-safe annotations**: `informix-driver` ships with `py.typed`
- **Python 3.12+ support**
- **Pipelined `executemany`**: 1.6× faster than IfxPy's per-row implementation
- **Rows addressable by name**: `row["col"]` and `row.col`, not just `row[0]`
## Reading rows by name
IfxPy gives you positional access and nothing else, which is why most
IfxPy codebases grow a helper like this:
```python
cols = [c[0].lower() for c in cur.description]
row_dict = dict(zip(cols, cur.fetchone()))
```
That gets you a dict and still no attribute access. Ask for `Row`
instead and all three work at once:
```python
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur.execute("SELECT config_key, config_value FROM settings")
row = cur.fetchone()
row[0], row["config_key"], row.config_key
```
Set on the connection, so it applies to every cursor from it. The
`.lower()` in the hand-written version is already a no-op, incidentally:
Informix folds unquoted identifiers to lower case.
It is opt-in because it costs about 9% on bulk fetch. See
[the API reference](/reference/api/#rows) for the numbers and the
edge cases.
## Migrating incrementally
If your codebase has hundreds of IfxPy call sites, you can do a partial migration:
1. Replace connection construction with `informix-driver` at the application boundary.
2. Where you used `IfxPy.fetch_assoc` (returns dict), wrap our cursor with a small adapter:
```python
def fetch_assoc(cur):
row = cur.fetchone()
if row is None:
return False
return dict(zip([c[0] for c in cur.description], row))
```
3. For `IfxPy.exec_immediate`, use `cur.execute(sql)` with no params.
4. For `IfxPy.bind_param`, use the params arg of `execute()`: `cur.execute(sql, (a, b, c))`.
Most application code can be migrated with `sed`-level transformations.

View File

@ -0,0 +1,71 @@
---
title: Use the connection pool
description: Sync and async connection pools, covering sizing, timeouts, lifecycle, and threading.
sidebar:
order: 2
---
import { Aside } from '@astrojs/starlight/components';
The connection pool amortizes the ~11 ms login handshake across many queries and gives you a thread-safe / task-safe acquire-release API. Use it any time the same process makes more than a handful of queries.
## Sync pool
```python
import informix_db
pool = informix_db.create_pool(
host="db.example.com", port=9088,
user="informix", password="...",
database="mydb", server="informix",
min_size=2, # warm up at least 2 connections at create time
max_size=10, # hard cap; acquires beyond this block
acquire_timeout=5.0, # raise PoolTimeout if no connection in 5s
max_idle=600.0, # close connections idle longer than 10 min
)
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT id, name FROM users WHERE id = ?", (42,))
print(cur.fetchone())
pool.close()
```
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically, so you never see a dirty connection from `pool.connection()`.
## Async pool
```python
import asyncio
from informix_db import aio
async def main():
pool = await aio.create_pool(
host="db.example.com",
user="informix", password="...",
database="mydb",
min_size=2, max_size=10,
)
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
print(await cur.fetchone())
await pool.close()
asyncio.run(main())
```
Same semantics, `async`-aware. Cancellation is safe here too: a cancelled task does not leak an in-flight worker onto a recycled connection.
## Sizing
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes", usually 816 for typical web/API services.
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`**. The server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
## Threading
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally, since `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but you should still give each thread its own connection. The lock is a backstop, not a license.

View File

@ -0,0 +1,83 @@
---
title: BLOB / CLOB read & write
description: Reading and writing smart-LOB columns end-to-end in pure Python.
sidebar:
order: 6
---
import { Aside } from '@astrojs/starlight/components';
`informix-driver` reads and writes smart-LOB columns (BLOB and CLOB) end-to-end without any native machinery. The implementation uses Informix's `lotofile` and `filetoblob` SQL functions, intercepted at the `SQ_FILE` (98) wire-protocol level.
## Reading a BLOB
```python
data: bytes = cur.read_blob_column(
"SELECT data FROM photos WHERE id = ?",
(42,),
)
```
`read_blob_column` returns the raw bytes. For very large BLOBs (multi-GB), see the streaming variant below.
## Writing a BLOB
```python
cur.write_blob_column(
"INSERT INTO photos VALUES (?, BLOB_PLACEHOLDER)",
blob_data=jpeg_bytes,
params=(42,),
)
```
The `BLOB_PLACEHOLDER` token in the SQL marks where the BLOB data goes. Other parameters are bound positionally to `params=`.
## Reading a CLOB
CLOBs use the same methods as BLOBs, and there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
```python
raw: bytes = cur.read_blob_column(
"SELECT body FROM articles WHERE id = ?",
(42,),
)
text: str = raw.decode("iso-8859-1") # or your DB_LOCALE's codec
```
Returning `bytes` rather than `str` is deliberate: the driver transports the LOB over the `SQ_FILE` channel and never sees the column's declared character set, so guessing an encoding here would be exactly the kind of silent assumption that produces mojibake in one deployment and works fine in another.
## Writing a CLOB
Pass `clob=True` so the write routes through `filetoclob` rather than `filetoblob`, and encode the text yourself:
```python
cur.write_blob_column(
"INSERT INTO articles VALUES (?, BLOB_PLACEHOLDER)",
"long article text… café résumé".encode("iso-8859-1"),
(42,),
clob=True,
)
```
The placeholder token is `BLOB_PLACEHOLDER` for both BLOB and CLOB writes.
## Server-side prerequisites
<Aside type="caution">
Smart-LOBs require server-side configuration that the IBM Developer Edition Docker image doesn't ship with by default:
- An **`sbspace`** must be created (`onspaces -c -S sbspace1 -p ...`)
- `SBSPACENAME` must be set in `$ONCONFIG`
- A **level-0 archive** must be taken (`ontape -s -L 0`) before BLOBs can be created
- The database must be created **with logging** (`CREATE DATABASE foo WITH LOG`)
Full setup commands are in [`docs/DECISION_LOG.md` §10](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md).
</Aside>
## How it works (briefly)
The `lotofile` server function returns a smart-LOB descriptor as a regular result column when called via `SELECT`. The driver intercepts the `SQ_FILE` (PDU 98) response that contains the LOB bytes and reassembles them client-side.
Writing reverses the flow: `filetoblob` is invoked via a placeholder pattern in the SQL, the driver sends the bytes via `SQ_FILE` PDUs, and the server stores them in the sbspace.
The architectural pivot from the heavier `SQ_FPROUTINE` + `SQ_LODATA` stack to this lighter `SQ_FILE` intercept is documented in [Phase 10/11 of the decision log](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/DECISION_LOG.md). The result is roughly 3× smaller than originally projected.

View File

@ -0,0 +1,55 @@
---
title: Connect with TLS
description: TLS-listener configuration, bring-your-own SSL context, and dev-mode self-signed handling.
sidebar:
order: 1
---
import { Aside } from '@astrojs/starlight/components';
Informix uses **dedicated TLS-enabled listener ports** (configured server-side in `sqlhosts`) rather than STARTTLS upgrade. Point `port` at the TLS listener (typically `9089`) when `tls` is enabled.
## Production: bring your own SSL context
```python
import ssl
import informix_db
ctx = ssl.create_default_context(cafile="/path/to/ca.pem")
# Optional: client cert auth
# ctx.load_cert_chain(certfile="/path/to/client.pem", keyfile="/path/to/client.key")
conn = informix_db.connect(
host="db.example.com",
port=9089,
user="informix",
password="...",
database="mydb",
server="informix",
tls=ctx,
)
```
Bring-your-own context is the recommended production pattern, giving you full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
## Dev / self-signed: tls=True
```python
informix_db.connect(host="127.0.0.1", port=9089, ..., tls=True)
```
`tls=True` is a convenience for development. It builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
## Server-side configuration
The Informix server needs a TLS listener entry in `sqlhosts`:
```
informix_tls onsoctcp myhost 9089
```
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089`, with no configuration needed.
<Aside type="tip">
If your connection hangs at the handshake, you've probably pointed at the non-TLS port (`9088` instead of `9089`). The non-TLS listener will accept the TCP connection but won't speak TLS, so the handshake stalls.
</Aside>

View File

@ -0,0 +1,92 @@
---
title: informix-driver
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket, with no CSDK, no JVM, and no native libraries.
template: splash
hero:
tagline: ''
editUrl: false
lastUpdated: false
next: false
prev: false
---
import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
<div class="ifx-features">
<div class="ifx-feature">
<Icon name="rocket" class="ifx-feature__icon" />
<h3>1.6× faster bulk inserts than IfxPy</h3>
<p>Pipelined <code>executemany</code> sends every BIND+EXECUTE PDU before draining responses. IBM's C driver still pays one round-trip per row. We figured we could do better.</p>
</div>
<div class="ifx-feature">
<Icon name="seti:python" class="ifx-feature__icon" />
<h3>~10% behind IfxPy on bulk fetches</h3>
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics, for now.</p>
</div>
<div class="ifx-feature">
<Icon name="puzzle" class="ifx-feature__icon" />
<h3>50 KB wheel. Zero native deps.</h3>
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.103.14, including the versions IfxPy doesn't.</p>
</div>
<div class="ifx-feature">
<Icon name="sun" class="ifx-feature__icon" />
<h3>Async, like a modern driver should be</h3>
<p>FastAPI, aiohttp, asyncio. Pool, connections, cursors all have <code>async def</code> versions. IfxPy has none of this.</p>
</div>
<div class="ifx-feature">
<Icon name="approve-check" class="ifx-feature__icon" />
<h3>PEP 249, no surprises</h3>
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy, plus threadsafe sharing through a per-connection wire lock.</p>
</div>
<div class="ifx-feature">
<Icon name="document" class="ifx-feature__icon" />
<h3>Clean-room reverse engineering</h3>
<p>Decompiled IBM JDBC. Annotated <code>socat</code> captures. Differential testing against IfxPy on every codec path. Every architectural decision lives in <a href="/explain/phase-log/">the phase log</a>. Receipts.</p>
</div>
</div>
## A query, end to end
```python
import informix_db
with informix_db.connect(
host="db.example.com", port=9088,
user="informix", password="...",
database="mydb", server="informix",
) as conn:
cur = conn.cursor()
cur.execute("SELECT id, name FROM users WHERE id = ?", (42,))
user_id, name = cur.fetchone()
```
That's it. No `IBM_DB_HOME`. No DSN file. No `libcrypt.so.1`.
## WTF did you build this for?
The existing tools were not my style.
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That means all of them: `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9). It needs four `LD_LIBRARY_PATH` entries. It needs `setuptools < 58`. And IfxPy itself is broken on Python 3.12+. For containerized deployments, ETL pipelines, FastAPI services, or anywhere a build toolchain on the runtime is friction, this driver is the alternative that didn't previously exist. Now it does.
## Read next
<CardGrid>
<Card title="Install & first query" icon="rocket">
Up and running with the IBM Informix Developer Edition Docker image in five minutes.
[Get started →](/start/quickstart/)
</Card>
<Card title="Compared to IfxPy" icon="random">
Head-to-head benchmarks, install gauntlet, and when each driver wins.
[Read →](/start/vs-ifxpy/)
</Card>
<Card title="The buffered reader" icon="information">
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1×, and the architectural mistake the first pass got wrong.
[Read →](/explain/buffered-reader/)
</Card>
<Card title="Architecture" icon="puzzle">
SQLI on the wire. Sockets, framing, codec, resultsets. How the layers stack.
[Read →](/explain/architecture/)
</Card>
</CardGrid>

View File

@ -0,0 +1,214 @@
---
title: API surface
description: Module-level functions, Connection / Cursor / Pool APIs, async equivalents.
sidebar:
order: 1
---
The top-level surface of `informix_db` and `informix_db.aio`. For full PEP 249 details, see the standard's [DB-API 2.0 spec](https://peps.python.org/pep-0249/).
## Module-level
```python
import informix_db
informix_db.connect(...) -> Connection
informix_db.create_pool(...) -> Pool
informix_db.apilevel # "2.0"
informix_db.threadsafety # 1
informix_db.paramstyle # "numeric"
```
```python
from informix_db import aio
await aio.connect(...) -> aio.Connection
await aio.create_pool(...) -> aio.Pool
```
## connect()
```python
informix_db.connect(
*,
host: str,
port: int = 9088,
user: str,
password: str,
database: str | None,
server: str, # DBSERVERNAME (not hostname)
autocommit: bool = False,
connect_timeout: float | None = None,
read_timeout: float | None = None,
keepalive: bool = False,
client_locale: str = "en_US.8859-1",
env: dict[str, str] | None = None,
tls: bool | ssl.SSLContext = False,
) -> Connection
```
## Connection
| Method / property | Description |
|---|---|
| `cursor(scrollable=False)` | Returns a new `Cursor`. Only one scrollable cursor may be open per connection; see below. |
| `commit()` | Commits the current transaction. |
| `rollback()` | Rolls back the current transaction. |
| `close()` | Closes the connection. Idempotent. |
| `fast_path_call(routine, *args)` | Direct UDF/SPL invocation, bypassing PREPARE/EXECUTE/FETCH. |
| `encoding` | Resolved Python codec for `client_locale`. |
| `closed` | `True` after `close()`. |
| `server_version` | Server release, e.g. `'…Version 12.10.FC6'`. Costs one query on first access, then cached. |
| `server_version_internal` | Raw login-response version string. Free. See note below. |
| `server_capabilities` | Negotiated `ServerCapabilities`, or `None` if the reply couldn't be decoded. |
`Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit.
There is no `conn.transaction()` helper and no `conn.autocommit` attribute. Earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
```python
conn = informix_db.connect(..., autocommit=False)
try:
cur = conn.cursor()
cur.execute("UPDATE accounts SET balance = balance - ? WHERE id = ?", (100, 1))
cur.execute("UPDATE accounts SET balance = balance + ? WHERE id = ?", (100, 2))
conn.commit()
except Exception:
conn.rollback()
raise
```
:::caution[One statement per session]
Informix gives a session a single statement slot. A non-scrollable cursor never trips over this, because it materializes its rows and releases the statement before `execute()` returns. A **scrollable** cursor holds the slot open on purpose, so while one is open, any other statement on that same connection raises `ProgrammingError`:
```python
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT id, body FROM big_table ORDER BY id")
scroll.fetch_first()
other = conn.cursor()
other.execute("SELECT COUNT(*) FROM audit") # ProgrammingError
```
Close the scrollable cursor first, or use a second connection for the other query. Re-running `execute()` on the scrollable cursor itself is fine; it closes its own server-side cursor first.
Before `2026.09.02` this was not refused. The server answered the second statement with `-285` and destroyed the scrollable cursor as well, whose next fetch then returned `-267`, "the transaction has been rolled back". Two errors, neither naming the cause.
:::
:::note[Transaction control written as SQL]
`commit()` and `rollback()` are the supported way to end a transaction, but `cursor.execute("BEGIN WORK")` and its `COMMIT` / `ROLLBACK` counterparts work too, and the connection tracks them from `2026.09.02` onward.
That matters because it used not to. Under `autocommit=True`, a SQL `BEGIN WORK` opened a real transaction the connection never learned about, and since `rollback()` is guarded by that state, **it returned successfully having sent nothing**. The rows it was asked to discard survived, and a pooled connection went back into circulation still holding the transaction and its locks.
:::
:::note[Two version numbers, and why]
The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect.
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')`. That is one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
:::
## Rows
Fetch methods return plain tuples by default. Pass `row_factory` to get rows that also answer to a column name and an attribute, the shape `pyodbc` and `mssql-python` provide:
```python
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur = conn.cursor()
cur.execute("SELECT tabid, tabname FROM systables")
row = cur.fetchone()
row[0] # 1
row["tabname"] # 'systables'
row.tabname # 'systables'
```
Set on the connection, so it applies to every cursor from it. A cursor can override it with `cur.row_factory = informix_db.Row`. Pools and the async API forward it unchanged.
`Row` subclasses `tuple`, so `row == (1, "systables")` is still true and anything treating rows as sequences keeps working. Slices return plain tuples, since a slice has no column map. `row.keys()` gives the column names and `row._asdict()` gives a dict.
:::note[Why it is opt-in]
On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` becomes a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript.
Defaulting it on would move the [1.05-1.15x ratio against IfxPy](/start/vs-ifxpy/) to roughly 1.15-1.25x. That is a good trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is yours to make.
:::
Four behaviours worth knowing:
- Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`. The `.lower()` people add by hand when building this themselves is already a no-op.
- Expression columns get server-generated names like `(count(*))`, which cannot be Python attributes. Those are reachable by subscript only.
- Duplicate column names resolve to the first occurrence, matching `pyodbc`. Positional access still reaches both.
- **A column beats a method of the same name.** `tuple` defines `count` and `index`; `Row` adds `keys`, `_asdict` and `_fields`. A column with any of those names wins, because otherwise it would hand back a bound method instead of a value. The methods stay reachable as `tuple.count(row, x)` and `Row.keys(row)`.
## Cursor
| Method / property | Description |
|---|---|
| `execute(sql, params=())` | Prepare + execute. Returns the cursor. |
| `executemany(sql, seq_of_params)` | Pipelined batch execute. |
| `fetchone()` | One row tuple, or `None`. |
| `fetchmany(size=arraysize)` | List of row tuples. |
| `fetchall()` | All remaining rows. |
| `scroll(value, mode="relative")` | Scrollable cursor positioning. Needs `cursor(scrollable=True)`. |
| `fetch_first()` / `fetch_last()` | Jump to the first / last row. Scrollable cursors only. |
| `fetch_prior()` / `fetch_relative(n)` / `fetch_absolute(n)` | Relative and absolute positioning. Scrollable cursors only. |
| `read_blob_column(sql, params)` | Read a BLOB **or** CLOB column → `bytes`. |
| `write_blob_column(sql, data, params, clob=False)` | Write a BLOB column; pass `clob=True` for CLOB. |
| `close()` | Closes the cursor + releases server resources. |
| `closed` | `True` after `close()`. |
| `description` | Sequence of column descriptors per PEP 249. |
| `rowcount` | Affected row count for DML; `-1` for SELECT. |
| `rownumber` | Current 0-indexed position, or `None` before the first row. |
| `arraysize` | Default `fetchmany()` size. |
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute. Earlier versions of this page listed all three, and none have existed.
CLOBs go through the BLOB methods. `read_blob_column` returns `bytes` either way, so decode it yourself with the column's encoding:
```python
cur.write_blob_column(
"INSERT INTO docs VALUES (?, BLOB_PLACEHOLDER)",
"café résumé".encode("iso-8859-1"),
(1,),
clob=True,
)
raw = cur.read_blob_column("SELECT txt FROM docs WHERE id = ?", (1,))
text = raw.decode("iso-8859-1")
```
For a server-assigned `SERIAL` after an INSERT, ask the server. Informix exposes it through `DBINFO`:
```python
cur.execute("INSERT INTO people (name) VALUES (?)", ("ada",))
cur.execute("SELECT DBINFO('sqlca.sqlerrd1') FROM systables WHERE tabid = 1")
new_id = cur.fetchone()[0]
```
## Pool
| Method | Description |
|---|---|
| `connection()` | Context manager yielding a connection from the pool. |
| `close()` | Closes all idle connections; waits for in-use to return. |
| `closed` | `True` after `close()`. |
| `size` | Current pool size. |
| `idle_count` | Connections currently idle in the pool. |
`aio.Pool` is identical except its `connection()` is an async context manager and `close()` is `async`.
## Exceptions
```
Error
├── Warning
└── DatabaseError
├── InterfaceError
├── DataError
├── OperationalError
│ └── PoolTimeout
├── IntegrityError
├── InternalError
├── ProgrammingError
└── NotSupportedError
```
See [Exceptions & error codes](/reference/exceptions/) for the SQLCODE → exception mapping.

View File

@ -0,0 +1,76 @@
---
title: Performance baselines
description: Single-connection benchmark results for codec, framing, end-to-end queries, and IfxPy.
sidebar:
order: 5
---
import { Aside } from '@astrojs/starlight/components';
These are the steady-state numbers for the current release (`2026.05.05.12`), measured against the IBM Informix Developer Edition Docker container on loopback.
<Aside type="note">
All numbers are **median over 10+ rounds** (pytest-benchmark `--rounds 10`). Reproduce with `make bench` from the repo root.
</Aside>
## Codec micro-benchmarks
| Operation | Mean | Throughput |
|---|---:|---:|
| `decode(int)` per cell | 139 ns | 7.2M ops/sec |
| `decode(varchar)` per cell | 280 ns | 3.6M ops/sec |
| `decode(decimal)` per cell | 410 ns | 2.4M ops/sec |
| `parse_tuple_payload` per row (5 cols) | 1.4 µs | 715K rows/sec |
## End-to-end
| Operation | Mean | Throughput |
|---|---:|---:|
| `SELECT 1` round-trip | ~140 µs | ~7K queries/sec |
| 1000-row SELECT | ~1.0 ms | ~990K rows/sec sustained |
| `executemany(1000)` in transaction | 32 ms | ~31,000 rows/sec |
| Pool acquire + query + release | 295 µs | ~3.4K queries/sec |
| Cold connect (login handshake) | 11 ms | ~90 connections/sec |
## vs IfxPy 3.0.5
| Benchmark | IfxPy | informix-driver | Result |
|---|---:|---:|---:|
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
| Cold connect | 11.0 ms | 10.5 ms | comparable |
| `executemany(1k)` | 23.5 ms | 23.2 ms | tied |
| `executemany(10k)` | 259 ms | **161 ms** | **informix-driver 1.6× faster** |
| `executemany(100k)` | 2376 ms | **1487 ms** | **informix-driver 1.6× faster** |
| `SELECT 1k` | 1.34 ms | 1.72 ms | IfxPy 1.28× |
| `SELECT 10k` | 11.7 ms | 16.1 ms | IfxPy 1.07× |
| `SELECT 100k` | 116 ms | 169 ms | IfxPy 1.15× |
For the methodology, IQR caveats, and reproduction instructions, see [Compared to IfxPy](/start/vs-ifxpy/).
## Phase progression on bulk SELECT
| Phase | 100k-row SELECT | Ratio vs IfxPy |
|---|---:|---:|
| Phase 36 | 280 ms | 2.40× slower |
| Phase 37 (per-column readers) | 250 ms | 2.10× slower |
| Phase 38 (codegen-inlined decoders) | 237 ms | 2.04× slower |
| **Phase 39 (connection-scoped buffered reader)** | **169 ms** | **1.15× slower** |
The Phase 39 jump is documented in [The buffered reader](/explain/buffered-reader/).
## Reproducing
```bash
git clone https://git.supported.systems/warehack.ing/informix-db
cd informix-db
make ifx-up # starts the dev container
make bench # runs all benchmarks
make compare # head-to-head vs IfxPy (handles IfxPy's install gauntlet)
```
For just the bulk-fetch progression:
```bash
pytest -m benchmark tests/benchmarks/test_select_scaling.py
```

View File

@ -0,0 +1,67 @@
---
title: Configuration & env flags
description: Runtime environment variables, connection arguments, pool tunables.
sidebar:
order: 3
---
## Environment variables
| Variable | Default | Effect |
|---|---|---|
| `IFX_BUFFERED_READER` | `1` | Enable connection-scoped read buffer (Phase 39). Set to `0` to disable. |
| `IFX_DEBUG_WIRE` | unset | When set to a truthy value, log wire-level PDU framing to stderr. Verbose; for debugging only. |
| `IFX_PROTOCOL_TRACE` | unset | When set to a path, write annotated wire captures to the file. |
| `IFX_DISABLE_PIPELINE` | unset | Disables pipelined `executemany` (Phase 33). Use only to A/B-measure. |
| `INFORMIXSERVER` | *(none)* | Read by `connect()` if `server=` is not provided. |
Environment variables are read at connection construction. Changing them at runtime doesn't affect existing connections.
## Connection arguments
Full keyword list for `informix_db.connect()`:
| Arg | Type | Default | Notes |
|---|---|---|---|
| `host` | `str` | required | TCP host. |
| `port` | `int` | `9088` | TCP port. `9089` is the typical TLS listener. |
| `user` | `str` | required | |
| `password` | `str` | required | |
| `database` | `str \| None` | required | `None` logs in without selecting a DB. |
| `server` | `str` | required | DBSERVERNAME, not hostname. |
| `autocommit` | `bool` | `False` | |
| `connect_timeout` | `float \| None` | `None` | TCP+login timeout. |
| `read_timeout` | `float \| None` | `None` | Per-read timeout. |
| `keepalive` | `bool` | `False` | `SO_KEEPALIVE`. |
| `client_locale` | `str` | `"en_US.8859-1"` | See [SQL ↔ Python types](/reference/types/) for codec mapping. |
| `env` | `dict[str, str] \| None` | `None` | Server-side session env. |
| `tls` | `bool \| ssl.SSLContext` | `False` | See [Connect with TLS](/how-to/tls/). |
## Pool tunables
| Arg | Default | Effect |
|---|---|---|
| `min_size` | `1` | Connections to pre-warm at create time. |
| `max_size` | `10` | Hard cap. |
| `acquire_timeout` | `30.0` | Raise `PoolTimeout` after this many seconds. |
| `max_idle` | `600.0` | Close connections idle longer than this (seconds). |
| `health_check` | `True` | Validate idle connections before returning from the pool. |
All other connection arguments are forwarded to each pool-created connection.
## Server-side session env
The `env={}` parameter of `connect()` sets server session variables sent in the login PDU:
```python
informix_db.connect(
...,
env={
"OPT_GOAL": "-1", # optimize for first-row return
"OPTOFC": "1", # auto-free cursors at fetch-close
"IFX_AUTOFREE": "1",
},
)
```
`CLIENT_LOCALE` is set automatically from `client_locale=`, so don't put it in `env=`.

View File

@ -0,0 +1,68 @@
---
title: Exceptions & error codes
description: PEP 249 exception hierarchy, Informix SQLCODE → Python exception mapping.
sidebar:
order: 4
---
The exception hierarchy is per PEP 249. All exceptions live in `informix_db` and inherit from `informix_db.Error`.
## Hierarchy
```
Error
├── Warning
└── DatabaseError
├── InterfaceError
├── DataError
├── OperationalError
│ └── PoolTimeout
├── IntegrityError
├── InternalError
├── ProgrammingError
└── NotSupportedError
```
## When each is raised
| Exception | Typical cause |
|---|---|
| `InterfaceError` | Misuse of the driver API itself (e.g. fetch on a closed cursor). |
| `DataError` | Type conversion failures, codec errors. |
| `OperationalError` | Network errors, timeouts, server unavailable, login failures. |
| `IntegrityError` | Constraint violations (PK, FK, unique, NOT NULL). |
| `InternalError` | Driver-internal invariant violation (file a bug). |
| `ProgrammingError` | Bad SQL syntax, missing tables, parameter binding errors. |
| `NotSupportedError` | Driver doesn't support the requested operation. |
| `PoolTimeout` | Pool acquire exceeded `acquire_timeout`. |
## SQLCODE mapping
Informix returns SQLCODE values; the driver maps them to the appropriate exception. A few common ones:
| SQLCODE | Meaning | Exception |
|---|---|---|
| `-201` | Syntax error | `ProgrammingError` |
| `-206` | Table not found | `ProgrammingError` |
| `-239` / `-268` / `-691` / `-703` | Unique / FK / NOT NULL violation | `IntegrityError` |
| `-329` | Database not found | `OperationalError` |
| `-908` | Connection terminated by server | `OperationalError` |
| `-1820` | Codeset conversion failure | `DataError` |
| `-908` / network errors | Server unavailable | `OperationalError` |
The full mapping lives in `src/informix_db/_errcodes.py`.
## Inspecting errors
Every exception carries the original SQLCODE and message:
```python
try:
cur.execute("INSERT INTO users VALUES (?, ?)", (1, "alice"))
except informix_db.IntegrityError as e:
print(e.sqlcode) # -239 (unique violation)
print(e.isam_code) # secondary error code
print(str(e)) # human-readable
```
`sqlcode` is the primary error; `isam_code` is the underlying ISAM error (when applicable). For multi-statement transactions, use `e.statement` to see which statement failed.

View File

@ -0,0 +1,121 @@
---
title: SQL ↔ Python types
description: Mapping table between Informix SQL types and Python types, with notes on edge cases.
sidebar:
order: 2
---
| SQL type | Python type | Notes |
|---|---|---|
| `SMALLINT` / `INT` / `SERIAL` | `int` | Arbitrary precision on the Python side. |
| `BIGINT` / `BIGSERIAL` | `int` | 8-byte two's complement on the wire. |
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer, a different wire format from `BIGINT` rather than an alias. See below. |
| `FLOAT` / `SMALLFLOAT` | `float` | IEEE 754 double / single. |
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | Exact precision preserved. |
| `CHAR` / `NCHAR` | `str` | Fixed width, space-padded; trailing spaces stripped on decode. |
| `VARCHAR` / `NVCHAR` / `LVARCHAR` | `str` | Variable length. Decoded using `client_locale`. |
| `BOOLEAN` | `bool` | |
| `DATE` | `datetime.date` | |
| `DATETIME YEAR TO …` | `datetime.datetime` / `datetime.time` / `datetime.date` | The Python type depends on the field range. |
| `INTERVAL DAY TO FRACTION` | `datetime.timedelta` | |
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type, because `datetime.timedelta` can't represent year-month intervals. |
| `BYTE` / `TEXT` (legacy in-row blobs) | `bytes` / `str` | |
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators, 72 bytes. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string, **not** decomposed into fields. See below. |
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema, **not** iterable. See below. |
| `NULL` | `None` | |
## INT8 and SERIAL8 are not BIGINT
Informix has two unrelated 64-bit integer types and they share nothing on the wire:
| | Wire format |
|---|---|
| `BIGINT` / `BIGSERIAL` (type 52 / 53) | 8 bytes, two's complement, big-endian |
| `INT8` / `SERIAL8` (type 17 / 18) | 10 bytes, sign-magnitude, halves stored high-last |
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `n` with *identical* magnitude bytes and the sign in a leading word. Decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
`INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived.
:::caution[Upgrade if you use these types]
`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it.
`2026.08.31` fixed two more in `LVARCHAR` framing: a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
`2026.09.02` fixed two more, both found by a new check that verifies each row consumed exactly its own payload. Smart LOBs were read as a flat 72-byte field when they actually occupy 149 bytes populated and 5 when NULL, so a `BLOB` or `CLOB` anywhere but the **last** column shifted every column after it. A NULL `SET` / `MULTISET` / `LIST` / `ROW` skipped its length field and did the same. That release also made a mismatch raise where it happens, naming the column and the byte delta, instead of handing back plausible-looking wrong values.
`2026.09.03` fixed one on the **write** path rather than the read path, which makes it the only entry here that affects data already stored. The `:N` placeholder rewriter was a regular expression and could not see a string literal, so it rewrote the inside of one: `'http://host:8080/x'` was stored as `'http://host?/x'`. Any `HH:MM` time, URL with a port, or `key:value` string was affected, whichever placeholder style you used. If you have written such values through this driver, they are worth checking.
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe. If you select a `BLOB` or `CLOB` in any position other than last, or a nullable collection column, nothing before `2026.09.02` is safe. Upgrade.
:::
## LVARCHAR wire framing
Worth knowing if you're reading captures. Every Informix server we've tested describes `LVARCHAR` columns as UDTVAR (type 40, `extended_name='lvarchar'`), and the value arrives wrapped in a UDT envelope:
```
[1-byte null indicator][4-byte length][content]
```
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte, since both carry a length of zero.
## DATETIME field ranges
Informix's `DATETIME YEAR TO X` is field-range typed. The Python type returned depends on which fields are present:
| Range | Python type |
|---|---|
| `YEAR TO YEAR` through `YEAR TO DAY` | `datetime.date` |
| `YEAR TO HOUR` through `YEAR TO FRACTION(5)` | `datetime.datetime` |
| `HOUR TO HOUR` through `HOUR TO FRACTION(5)` | `datetime.time` |
## Decimal precision
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly, with no float intermediate.
For binding `Decimal` values into INSERT/UPDATE, the driver uses the column's declared scale. Pass `Decimal` for exact values; `float` works but may cause rounding at the column scale.
## NULL
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL, because `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
## Type extensions
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count**, mirroring the server's own representation, with `years` and `remainder_months` available as derived properties:
```python
from informix_db import IntervalYM
ym = IntervalYM(27) # 2 years, 3 months
ym.years # 2
ym.remainder_months # 3
str(ym) # '2-03'
cur.execute("INSERT INTO contracts (term) VALUES (?)", (ym,))
cur.execute("SELECT term FROM contracts WHERE id = ?", (1,))
result = cur.fetchone()[0] # IntervalYM(months=27)
```
Negative intervals are supported; the sign lives on `months` and propagates to both derived properties.
## ROW and collection columns
`informix_db.RowValue` and `informix_db.CollectionValue` are returned for `ROW`, `SET`, `MULTISET`, and `LIST` columns. Both are **opaque wrappers, not decomposed values**:
```python
row_val.raw # bytes: the server's textual representation, e.g. b"ROW('Alice',30)"
row_val.schema # str: the column's declared schema
coll_val.raw # bytes, e.g. b'LIST{10,20,30}'
coll_val.kind # 'set' | 'multiset' | 'list' | 'collection'
coll_val.element_schema # str
```
They are not iterable and do not expose fields by name. Fully parsing a composite type means reimplementing JDBC's `IfxComplexInput`, which we haven't done.
If you need individual fields today, project them in SQL. The server does the decomposition for you and you get ordinary typed columns back:
```python
cur.execute("SELECT person.name, person.age FROM staff") # str, int
```

View File

@ -0,0 +1,166 @@
---
title: Install & first query
description: Five minutes from pip install to a real SELECT against a Dockerised Informix server.
sidebar:
order: 2
---
import { Tabs, TabItem, Steps, Aside } from '@astrojs/starlight/components';
This page gets you from a clean Python environment to a working query against a real Informix server in five minutes. We'll use the official IBM Informix Developer Edition Docker image so you don't need an Informix license.
## Prerequisites
- Python 3.10 or newer
- Docker, for the dev server (skip if you already have an Informix instance)
- 4 GB of free RAM for the dev container
## 1. Install the driver
<Tabs>
<TabItem label="uv (recommended)">
```bash
uv add informix-driver
```
</TabItem>
<TabItem label="pip">
```bash
pip install informix-driver
```
</TabItem>
<TabItem label="poetry">
```bash
poetry add informix-driver
```
</TabItem>
</Tabs>
That's the entire dependency. No system packages, no `LD_LIBRARY_PATH`, no `libcrypt.so.1`.
## 2. Start the dev container
```bash
docker run -d --name informix-dev \
-e LICENSE=accept \
-p 9088:9088 \
-p 9089:9089 \
--privileged \
icr.io/informix/informix-developer-database:15.0.1.0.3DE
```
The image takes ~90 seconds to initialize. Watch the logs until you see `oninit running`:
```bash
docker logs -f informix-dev
```
<Aside type="tip">
The `--privileged` flag is required by the dev image, which tries to manage shared memory limits. For production servers this isn't a thing.
</Aside>
## 3. Run your first query
Save this as `hello.py`:
```python
import informix_db
with informix_db.connect(
host="127.0.0.1",
port=9088,
user="informix",
password="in4mix",
database="sysmaster",
server="informix",
) as conn:
cur = conn.cursor()
cur.execute(
"SELECT FIRST 5 dbsname, tabname "
"FROM systables WHERE tabid > 99"
)
for row in cur.fetchall():
print(row)
```
Then:
```bash
python hello.py
```
You should see five rows from Informix's system catalog. If you do, congratulations: you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
## What just happened
When `informix_db.connect()` returned, the driver had:
1. Opened a TCP socket to `127.0.0.1:9088`
2. Sent an `SQ_INFO` PDU containing client capabilities (locale, byte order, app name)
3. Received the server's identification (`IBM Informix Dynamic Server Version 15.0.1.0.3`)
4. Negotiated authentication via `SQ_PASSWD`
5. Opened the `sysmaster` database via `SQ_DBOPEN`
6. Returned a `Connection` object ready for queries
`cur.execute()` sent an `SQ_PREPARE` PDU with your SQL, parsed the response into a column descriptor, sent `SQ_DESCRIBE`, then `SQ_OPEN` to start the cursor. `cur.fetchall()` issued `SQ_FETCH` PDUs and decoded each `SQ_TUPLE` payload via per-column readers.
If you want to see this happen byte-by-byte, [the architecture page](/explain/architecture/) walks through the wire protocol with annotated captures.
## 4. Try parameter binding
```python
with informix_db.connect(host="127.0.0.1", port=9088, user="informix",
password="in4mix", database="sysmaster",
server="informix") as conn:
cur = conn.cursor()
cur.execute(
"SELECT tabname FROM systables WHERE tabid = ?",
(1,),
)
print(cur.fetchone())
```
`?` and `:1` both work. Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
Rows come back as plain tuples. If you would rather read them by column
name, pass `row_factory=informix_db.Row` to `connect()` and you get
`row["name"]` and `row.name` alongside `row[0]`. See
[the API reference](/reference/api/#rows).
## 5. Use the connection pool
For real applications, prefer the pool:
```python
import informix_db
pool = informix_db.create_pool(
host="127.0.0.1", port=9088,
user="informix", password="in4mix",
database="sysmaster", server="informix",
min_size=2, max_size=10,
acquire_timeout=5.0,
)
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT 1 FROM systables WHERE tabid = 1")
print(cur.fetchone())
pool.close()
```
The pool is thread-safe and has a per-connection wire lock, so accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
## What's next
<Steps>
1. **Going async?** [Async with FastAPI →](/how-to/async-fastapi/) walks through a real FastAPI app with the async pool.
2. **Connecting to production?** [Connect with TLS →](/how-to/tls/) covers TLS-listener configuration and bring-your-own-context patterns.
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/), including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
4. **Migrating from IfxPy?** [Migrate from IfxPy →](/how-to/migrate-from-ifxpy/) covers the API differences and the things IfxPy does that we don't (yet).
</Steps>

View File

@ -0,0 +1,149 @@
---
title: Compared to IfxPy
description: Head-to-head benchmarks against IBM's C-bound Python driver. Where each one wins and why.
sidebar:
order: 3
---
import { Aside } from '@astrojs/starlight/components';
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver, a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Informix Developer Edition Docker container on the same host. Methodology and reproduction steps live in [`tests/benchmarks/compare/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare) in the repo.
## Headline numbers
| Benchmark | IfxPy 3.0.5 (C) | informix-driver (pure Python) | Result |
|---|---:|---:|---:|
| Single-row SELECT round-trip | 118 µs | 114 µs | comparable |
| ~10-row server-side query | 130 µs | 159 µs | IfxPy 22% faster |
| Cold connect (login handshake) | 11.0 ms | 10.5 ms | comparable |
| `executemany(1k)` in transaction | 23.5 ms | 23.2 ms | tied |
| **`executemany(10k)` in transaction** | 259 ms | **161 ms** | **informix-driver 1.6× faster** |
| **`executemany(100k)` in transaction** | 2376 ms | **1487 ms** | **informix-driver 1.6× faster** |
| `SELECT 1k` rows | 1.34 ms | 1.72 ms | IfxPy 1.28× faster |
| `SELECT 10k` rows | 11.7 ms | 16.1 ms | IfxPy 1.07× faster |
| `SELECT 100k` rows | 116 ms | 169 ms | IfxPy 1.15× faster |
<Aside type="note">
Phase 39's connection-scoped buffered reader closed the bulk-fetch gap from a steady ~2.4× to ~1.051.15×. The story of how that landed is in [the buffered reader page](/explain/buffered-reader/).
</Aside>
## When informix-driver wins
### Bulk inserts at scale
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms**, so `informix-driver` is 1.6× faster.
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row, and N RTTs at ~80 µs each adds up to the 100 ms gap.
```python
# Both drivers
cur.executemany(
"INSERT INTO orders VALUES (?, ?, ?)",
rows, # list of 10_000 tuples
)
# informix-driver: 161 ms (10k PDUs sent, then 10k responses drained)
# IfxPy: 259 ms (10k round-trips, each blocking on response)
```
### Containerized deployment
`informix-driver` ships as a 50 KB pure-Python wheel with **zero runtime dependencies**. Your Dockerfile is:
```dockerfile
FROM python:3.13-slim
RUN pip install informix-driver
```
IfxPy's deployment surface is dramatically larger:
- 92 MB IBM OneDB Client tarball
- `setuptools < 58` build pin
- `LD_LIBRARY_PATH` configuration for four directories
- `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9)
- C compiler in the build image
For slim images, multi-stage builds, FaaS deployments, or anywhere build-toolchain-on-the-runtime is friction, `informix-driver` is the only reasonable option.
### Modern Python
IfxPy works on Python ≤ 3.11 currently. The C extension breaks on 3.12+ (PyConfig changes, removed `_PyImport_AcquireLock`, etc.).
`informix-driver` works unmodified on **3.10, 3.11, 3.12, 3.13, and 3.14**. We've kept a CI matrix on every minor version since 3.10 from the start.
### Async
`informix-driver` ships an async API:
```python
from informix_db import aio
async def main():
pool = await aio.create_pool(...)
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT ...")
rows = await cur.fetchall()
```
IfxPy has no async support, so every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
## When IfxPy wins
### Large analytical fetches
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 515% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock, which is meaningful but not disqualifying.
The gap is closing phase by phase:
| Phase | Bulk-fetch ratio vs IfxPy |
|---|---|
| Phase 36 | 2.40× slower |
| Phase 37 (per-column readers) | 2.10× slower |
| Phase 38 (codegen-inlined decoders) | 2.04× slower |
| **Phase 39 (connection-scoped buffered reader)** | **1.15× slower** |
If you're running analytical reports that pull millions of rows in a single SELECT and the per-row decode overhead is a measurable cost, IfxPy may be marginally faster today. For most application workloads it isn't.
### Workloads built around CSDK extensions
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API. See [the migration guide](/how-to/migrate-from-ifxpy/).
## Methodology
Benchmarks are pytest-benchmark fixtures in `tests/benchmarks/compare/` against the official `icr.io/informix/informix-developer-database:15.0.1.0.3DE` image, running on the same loopback as the Python process.
Reported numbers are **median over 10+ rounds**, with IQR included. Why median over mean: the first round of any run includes JIT warmup, page-cache miss, and a TCP slow-start round-trip. The mean is contaminated by these one-shot costs in a way that misrepresents steady-state behavior. Median + IQR is what we report.
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise. A fair reading is "515% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
To reproduce:
```bash
git clone https://git.supported.systems/warehack.ing/informix-db
cd informix-db/tests/benchmarks/compare
make ifx-up
make compare
```
The `Makefile` handles the IfxPy install gauntlet (Python ≤ 3.11 environment, `setuptools < 58`, `libcrypt.so.1` symlink, OneDB CSDK download, the four `LD_LIBRARY_PATH` exports) so you don't have to learn it manually.
## Summary
Use `informix-driver` when:
- You're writing new code in Python ≥ 3.10
- Your workload is bulk-insert / ETL / log-shipping
- You want async / FastAPI integration
- You're deploying in containers or to Python environments where build toolchains are friction
- Your platform doesn't have `libcrypt.so.1`
Use IfxPy when:
- You have an existing IfxPy codebase
- You're running large analytical SELECTs and the 515% decode-side gap matters
- You're constrained to Python ≤ 3.11 anyway
For everything else, the cost-benefit favors `pip install informix-driver`.

View File

@ -0,0 +1,87 @@
---
title: WTF did you build this for?
description: A pure-Python Informix driver, why it didn't exist before, and what it's good for.
sidebar:
order: 1
label: WTF did you build this for?
---
The existing tools were not my style.
Every Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That covers `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
## The problem with IBM's C SDK
The IBM Informix Client SDK (CSDK), now packaged as part of OneDB Client, is a 92 MB tarball with a non-trivial install gauntlet:
- Python ≤ 3.11 (IfxPy is broken on 3.12+)
- `setuptools < 58` (legacy build system)
- Permissive `CFLAGS` for the C extension build
- Manual download of the 92 MB ODBC tarball
- Four `LD_LIBRARY_PATH` directories
- `libcrypt.so.1`, deprecated in 2018 and missing on Arch, Fedora 35+, RHEL 9
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db`; the distribution name dodges PyPI's 2008-vintage `informixdb` package, while the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
## What it does
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly. It is the same protocol IBM's JDBC driver uses, and the same one the CSDK speaks under the hood. No native code is in the thread of execution.
The wire protocol was reverse-engineered through three sources:
1. **Decompiled IBM JDBC driver** (`com.informix.jdbc.IfxConnection` and friends), used as a clean-room reference for PDU shapes and protocol semantics.
2. **Annotated `socat` captures** of real client/server traffic against the IBM Informix Developer Edition Docker image.
3. **Differential testing** against `IfxPy`, so every codec path is checked against the C driver's behavior on the same data.
The result is a PEP 249 compliant driver with a sync API, an async API (FastAPI / asyncio compatible), a connection pool, TLS support, smart-LOB read/write, scrollable cursors, fast-path stored procedure invocation, and bulk-insert / bulk-fetch performance within ~1060% of the C driver depending on workload.
## What it's good for
The places where `informix-driver` is unambiguously the right choice:
- **ETL and bulk-load pipelines.** Pipelined `executemany` (Phase 33) is 1.6× faster than IfxPy at scale because every BIND+EXECUTE PDU goes out before any responses are drained. IfxPy still pays one round-trip per `IfxPy.execute(stmt, tuple)` call.
- **Container deployments.** The 50 KB wheel and absent native deps mean a slim base image works. No multi-stage build to compile the CSDK.
- **Modern Python.** Works on 3.10 through 3.14 unmodified. IfxPy hasn't shipped 3.12 wheels.
- **Async / FastAPI.** Native async support via thread-pool wrapping. IfxPy is fully synchronous; using it from FastAPI requires `run_in_executor` boilerplate and gives up the connection pool's natural async semantics.
- **Anywhere `libcrypt.so.1` is missing.** Modern Linux distributions ship `libcrypt.so.2`. IfxPy refuses to load without `libcrypt.so.1`. We don't link against either.
## What IfxPy is still better at
Honesty matters here:
- **Large analytical fetches.** IfxPy's C-level `fetch_tuple` decoder is faster than our Python `parse_tuple_payload` (~1.1 µs/row vs ~2.0 µs/row after Phase 39). For workloads pulling 10k+ rows in a single SELECT where the per-row decode cost dominates, IfxPy is currently 515% faster. The gap is shrinking phase by phase.
- **Workloads built around the CSDK.** If your existing code already uses IfxPy idioms (`IfxPyDbi.connect_pooled`, IBM's specific cursor extensions), the migration to `informix-driver` is straightforward but not zero-cost.
The honest summary table from the [comparison page](/start/vs-ifxpy/):
| Workload | Winner | Margin |
|---|---|---|
| Bulk insert (`executemany` 10k100k rows) | `informix-driver` | 1.6× faster |
| Bulk SELECT (10k100k rows) | IfxPy | 1.051.15× faster |
| Single-row queries | tied | within noise |
| Cold connect | tied | within noise |
| Containerized deployment | `informix-driver` | no contest |
| Python 3.12+ | `informix-driver` | only option |
## Production-ready
Every finding from a system-wide failure-mode audit (data correctness, wire safety, resource leaks, concurrency, async cancellation) has been addressed:
- Pool no longer returns connections with open transactions
- Per-connection wire lock prevents PDU interleaving from accidental sharing
- Async cancellation cannot leak running workers onto recycled connections
- `_raise_sq_err` no longer masks wire desync via bare-except
- Cursor finalizers release server-side resources on mid-fetch raise
- 5 medium-severity hardening items resolved
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
470+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 472/472 on 15.0.1.0.3DE and 14.10.FC7W1DE, and 471/472 on 12.10.FC12W1DE (the single skip is a common table expression, which 12.10 predates). `make test-matrix` runs the lot.
That matrix exists because it turned out to be needed. A user reported corrupted result sets on Informix 12; the cause was three framing bugs that affected every version including the one we tested against, and they'd survived because no fixture used the affected types. Testing one server and inferring the rest is how that happens.
## Read next
- **[Install & first query →](/start/quickstart/)**: five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
- **[Compared to IfxPy →](/start/vs-ifxpy/)**: full head-to-head benchmarks, methodology, and reproduction.
- **[Architecture →](/explain/architecture/)**: how the layers stack, from socket through framing, codec, resultset, and cursor.

View File

@ -0,0 +1,428 @@
/* Component-scoped styles for the custom Hero and homepage modules */
.ifx-hero {
display: grid;
grid-template-columns: minmax(0, 1fr) minmax(0, 1.1fr);
gap: clamp(1.5rem, 4vw, 4rem);
align-items: center;
padding: clamp(2rem, 5vw, 4.5rem) 0 clamp(2rem, 4vw, 3.5rem);
border-bottom: 1px solid var(--sl-color-hairline-light);
}
@media (max-width: 800px) {
.ifx-hero {
grid-template-columns: 1fr;
}
}
.ifx-hero__eyebrow {
display: inline-flex;
align-items: center;
gap: 0.5rem;
font-family: var(--sl-font-mono);
font-size: 0.75rem;
letter-spacing: 0.18em;
text-transform: uppercase;
color: var(--sl-color-text-accent);
margin-bottom: 0.75rem;
}
.ifx-hero__eyebrow::before {
content: '';
width: 8px;
height: 8px;
background: var(--ifx-amber);
border-radius: 1px;
box-shadow: 0 0 12px var(--ifx-amber);
animation: ifx-pulse 2.4s ease-in-out infinite;
}
@keyframes ifx-pulse {
0%, 100% { opacity: 0.4; }
50% { opacity: 1; }
}
.ifx-hero__title {
font-size: clamp(2rem, 5vw, 3.25rem);
line-height: 1.05;
letter-spacing: -0.02em;
font-weight: 700;
margin: 0 0 1rem;
color: var(--sl-color-white);
}
.ifx-hero__title strong {
background: linear-gradient(180deg, var(--ifx-amber-bright) 0%, var(--ifx-amber) 100%);
-webkit-background-clip: text;
background-clip: text;
color: transparent;
font-weight: inherit;
}
.ifx-hero__lede {
font-size: clamp(1rem, 1.5vw, 1.15rem);
line-height: 1.55;
color: var(--sl-color-gray-2);
max-width: 36rem;
margin: 0 0 1.75rem;
}
.ifx-hero__cta {
display: flex;
flex-wrap: wrap;
gap: 0.75rem;
}
.ifx-hero__cta a {
display: inline-flex;
align-items: center;
gap: 0.4rem;
padding: 0.6rem 1.1rem;
border-radius: 6px;
font-weight: 500;
text-decoration: none !important;
transition: transform 0.08s ease, box-shadow 0.15s ease;
}
.ifx-hero__cta a:hover {
transform: translateY(-1px);
}
.ifx-hero__cta .primary {
background: var(--ifx-amber);
color: var(--ifx-charcoal-0);
}
.ifx-hero__cta .primary:hover {
box-shadow: 0 4px 16px rgba(245, 165, 36, 0.35);
}
.ifx-hero__cta .secondary {
border: 1px solid var(--sl-color-hairline-shade);
color: var(--sl-color-text);
background: transparent;
}
.ifx-hero__cta .secondary:hover {
border-color: var(--sl-color-accent);
}
.ifx-hero__install {
font-family: var(--sl-font-mono);
font-size: 0.95rem;
margin-top: 1.5rem;
padding: 0.75rem 1rem;
background: var(--sl-color-bg-inline-code);
border-left: 2px solid var(--ifx-amber);
border-radius: 0 4px 4px 0;
color: var(--sl-color-text);
user-select: all;
}
.ifx-hero__install::before {
content: '$ ';
color: var(--sl-color-text-accent);
user-select: none;
}
/* Wire-dump easter egg: types out real captured handshake bytes */
.ifx-wiredump {
font-family: var(--sl-font-mono);
font-size: 0.78rem;
line-height: 1.55;
background: var(--ifx-charcoal-0);
color: var(--ifx-amber);
border: 1px solid var(--sl-color-hairline);
border-radius: 8px;
padding: 1.25rem 1rem;
height: clamp(280px, 35vw, 360px);
/* x: auto allows hex content to scroll horizontally inside the dump
when the column is too narrow (tablet portrait, narrow desktop with
2-col hero); y: hidden keeps the dump's vertical bounds fixed so
unrevealed animation lines stay clipped below the fold */
overflow-x: auto;
overflow-y: hidden;
position: relative;
box-shadow: inset 0 0 60px rgba(245, 165, 36, 0.04);
}
:root[data-theme='light'] .ifx-wiredump {
background: #1a1612;
}
.ifx-wiredump__scroll {
white-space: pre;
height: 100%;
overflow: hidden;
}
.ifx-wiredump__line {
display: block;
opacity: 0;
white-space: pre;
}
.ifx-wiredump__line.is-visible {
opacity: 1;
}
.ifx-wiredump__byte--ascii {
color: #ffd884;
}
.ifx-wiredump__caret {
display: inline-block;
width: 8px;
height: 1em;
background: var(--ifx-amber);
vertical-align: text-bottom;
margin-left: 2px;
animation: ifx-blink 1.1s steps(2) infinite;
}
@keyframes ifx-blink {
0%, 50% { opacity: 1; }
51%, 100% { opacity: 0; }
}
.ifx-wiredump__direction {
color: var(--sl-color-gray-3);
margin-right: 0.5rem;
}
.ifx-wiredump__direction--out {
color: var(--ifx-amber);
}
.ifx-wiredump__direction--in {
color: #6dd2a4;
}
.ifx-wiredump__caption {
font-family: var(--sl-font-mono);
font-size: 0.7rem;
letter-spacing: 0.1em;
text-transform: uppercase;
color: var(--sl-color-gray-3);
margin-top: 0.6rem;
text-align: right;
}
/* Homepage feature grid */
.ifx-features {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
gap: 1.25rem;
margin: 3rem 0;
}
.ifx-feature {
padding: 1.5rem;
border: 1px solid var(--sl-color-hairline-light);
border-radius: 8px;
background: var(--sl-color-bg-nav);
transition: border-color 0.15s ease, transform 0.1s ease;
}
.ifx-feature:hover {
border-color: var(--sl-color-accent);
}
.ifx-feature__icon {
width: 28px;
height: 28px;
margin-bottom: 0.75rem;
color: var(--sl-color-text-accent);
}
.ifx-feature h3 {
font-size: 1.05rem;
margin: 0 0 0.5rem;
letter-spacing: -0.01em;
}
.ifx-feature p {
font-size: 0.92rem;
line-height: 1.5;
margin: 0;
color: var(--sl-color-gray-2);
}
/* Supported Systems "joint" badge, appears below every page's footer */
.ifx-ss-badge {
margin-top: 3.5rem;
padding: 0;
border-top: 1px solid var(--sl-color-hairline-light);
}
.ifx-ss-badge__link {
display: grid;
grid-template-columns: auto minmax(0, 1fr);
gap: 1.25rem;
align-items: center;
padding: 1.5rem 0 0.5rem;
text-decoration: none !important;
color: var(--sl-color-text);
transition: color 0.12s ease;
}
.ifx-ss-badge__link:hover {
color: var(--sl-color-text-accent);
}
.ifx-ss-badge__link:hover .ifx-ss-badge__cta {
gap: 0.5rem;
}
.ifx-ss-badge__logo {
width: 60px;
height: 45px;
flex-shrink: 0;
/* Slight lift so the punch-card pattern reads */
filter: drop-shadow(0 1px 2px rgba(37, 99, 235, 0.12));
}
.ifx-ss-badge__copy {
min-width: 0;
}
.ifx-ss-badge__heading {
font-family: var(--sl-font-mono);
font-size: 0.78rem;
font-weight: 600;
letter-spacing: 0.16em;
text-transform: uppercase;
color: var(--sl-color-text-accent);
margin: 0 0 0.4rem;
border: none !important;
padding: 0 !important;
}
.ifx-ss-badge__body {
font-size: 0.92rem;
line-height: 1.55;
color: var(--sl-color-gray-2);
margin: 0 0 0.5rem;
}
.ifx-ss-badge__name {
color: #3b82f6;
font-weight: 600;
}
:root[data-theme='light'] .ifx-ss-badge__name {
color: #2563eb;
}
.ifx-ss-badge__cta {
display: inline-flex;
align-items: center;
gap: 0.35rem;
font-size: 0.85rem;
font-weight: 500;
color: var(--sl-color-text-accent);
transition: gap 0.15s ease;
}
@media (max-width: 540px) {
.ifx-ss-badge__link {
grid-template-columns: 1fr;
gap: 0.85rem;
text-align: left;
}
.ifx-ss-badge__logo {
width: 50px;
height: 38px;
}
}
/* ============================================================
* Mobile (640px): tighten the hero, shrink the wire-dump,
* hide the ASCII column, prevent runaway horizontal overflow
* ============================================================ */
@media (max-width: 640px) {
/* Defensive guard against any descendant forcing horizontal scroll:
the wire-dump's white-space: pre content was overflowing the
hero column and propagating up to the page. overflow-x: hidden
on .ifx-hero contains it without affecting page-level scroll. */
.ifx-hero,
.ifx-hero__copy,
.ifx-hero__visual {
min-width: 0; /* allow flex/grid item to shrink below content min-width */
}
.ifx-hero {
padding: 1.25rem 0 1.75rem;
gap: 1.25rem;
}
.ifx-hero__eyebrow {
flex-wrap: wrap;
row-gap: 0.25rem;
font-size: 0.7rem;
letter-spacing: 0.12em;
}
.ifx-hero__title {
font-size: clamp(1.75rem, 7.5vw, 2.5rem);
line-height: 1.1;
}
.ifx-hero__lede {
font-size: 0.95rem;
line-height: 1.55;
}
.ifx-hero__cta {
gap: 0.5rem;
}
.ifx-hero__cta a {
padding: 0.55rem 0.9rem;
font-size: 0.92rem;
}
.ifx-hero__install {
font-size: 0.85rem;
word-break: break-all; /* survive any future longer install command */
}
/* The wire-dump is the biggest mobile risk: pre-formatted hex lines
are ~50 chars wide. Two things together make it readable on
narrow viewports: smaller font and ASCII column hidden. The
overflow-x: auto fallback already lives on the base rule. */
.ifx-wiredump {
font-size: 0.62rem;
height: clamp(220px, 60vw, 280px);
padding: 0.85rem 0.65rem;
}
.ifx-wiredump__byte--ascii {
display: none;
}
.ifx-wiredump__caption {
font-size: 0.62rem;
text-align: left;
}
.ifx-features {
gap: 0.9rem;
margin: 2rem 0;
}
.ifx-feature {
padding: 1.1rem;
}
.ifx-feature h3 {
font-size: 1rem;
}
}
/* Performance-bar visual for the comparison table */
.ifx-perfbar {
position: relative;
height: 6px;
background: var(--sl-color-bg-inline-code);
border-radius: 3px;
overflow: hidden;
margin-top: 0.25rem;
}
.ifx-perfbar__fill {
position: absolute;
inset: 0 auto 0 0;
background: linear-gradient(90deg, var(--ifx-amber) 0%, var(--ifx-amber-bright) 100%);
border-radius: 3px;
}

View File

@ -0,0 +1,137 @@
/*
* informix-driver docs theme
* - Charcoal base (no purple gradients, ever)
* - Amber accent, a CRT-monitor nod distinct from sibling sites' cyan
* - Inter for body, IBM Plex Mono for technical bytes
*/
:root {
--ifx-amber: #f5a524;
--ifx-amber-bright: #ffb84d;
--ifx-amber-dim: #b87a18;
--ifx-charcoal-0: #0e0d0c;
--ifx-charcoal-1: #161412;
--ifx-charcoal-2: #1f1c1a;
--ifx-charcoal-3: #2a2622;
--ifx-paper: #faf7f2;
--ifx-paper-2: #f3eee5;
--ifx-ink: #1a1612;
--ifx-ink-soft: #4a443c;
--ifx-rule: rgba(245, 165, 36, 0.18);
--sl-font: 'Inter', system-ui, -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
--sl-font-mono: 'IBM Plex Mono', ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace;
}
:root[data-theme='dark'] {
--sl-color-accent-low: #3a2810;
--sl-color-accent: var(--ifx-amber);
--sl-color-accent-high: var(--ifx-amber-bright);
--sl-color-white: #fbf7ee;
--sl-color-gray-1: #ece6da;
--sl-color-gray-2: #c8c1b1;
--sl-color-gray-3: #948b78;
--sl-color-gray-4: #5a5246;
--sl-color-gray-5: #2e2823;
--sl-color-gray-6: var(--ifx-charcoal-2);
--sl-color-black: var(--ifx-charcoal-0);
--sl-color-bg: var(--ifx-charcoal-0);
--sl-color-bg-nav: var(--ifx-charcoal-1);
--sl-color-bg-sidebar: var(--ifx-charcoal-1);
--sl-color-bg-inline-code: var(--ifx-charcoal-2);
--sl-color-bg-accent: var(--ifx-amber);
--sl-color-text: #ece6da;
--sl-color-text-accent: var(--ifx-amber-bright);
--sl-color-text-invert: var(--ifx-charcoal-0);
--sl-color-hairline: rgba(245, 165, 36, 0.14);
--sl-color-hairline-light: rgba(245, 165, 36, 0.08);
--sl-color-hairline-shade: rgba(245, 165, 36, 0.22);
}
:root[data-theme='light'] {
--sl-color-accent-low: #fbe8c8;
--sl-color-accent: var(--ifx-amber-dim);
--sl-color-accent-high: #7a4f0a;
--sl-color-white: #1a1612;
--sl-color-gray-1: #2e2823;
--sl-color-gray-2: #4a443c;
--sl-color-gray-3: #6e665a;
--sl-color-gray-4: #948b78;
--sl-color-gray-5: #d8d1c1;
--sl-color-gray-6: #ebe5d6;
--sl-color-gray-7: #f3eee5;
--sl-color-black: var(--ifx-paper);
--sl-color-bg: var(--ifx-paper);
--sl-color-bg-nav: var(--ifx-paper-2);
--sl-color-bg-sidebar: var(--ifx-paper-2);
--sl-color-bg-inline-code: #ece6d4;
--sl-color-bg-accent: var(--ifx-amber-dim);
--sl-color-text: var(--ifx-ink);
--sl-color-text-accent: var(--ifx-amber-dim);
--sl-color-text-invert: var(--ifx-paper);
--sl-color-hairline: rgba(120, 80, 12, 0.18);
--sl-color-hairline-light: rgba(120, 80, 12, 0.10);
--sl-color-hairline-shade: rgba(120, 80, 12, 0.28);
}
/* Tighten heading rhythm; Starlight defaults are a touch loose for technical docs */
.sl-markdown-content h2 {
margin-top: 2.5rem;
border-top: 1px solid var(--sl-color-hairline-light);
padding-top: 1.5rem;
letter-spacing: -0.01em;
}
.sl-markdown-content h3 {
margin-top: 1.75rem;
letter-spacing: -0.005em;
}
/* Inline code: monospace + amber tint, no jarring background block */
.sl-markdown-content :not(pre) > code {
font-feature-settings: 'ss02', 'cv02';
border: 1px solid var(--sl-color-hairline-shade);
font-size: 0.9em;
padding: 0.1em 0.35em;
border-radius: 4px;
}
/* Tables: dense, technical, amber column rules, for type-mapping & benchmark tables */
.sl-markdown-content table {
border-collapse: collapse;
font-variant-numeric: tabular-nums;
font-size: 0.92rem;
}
.sl-markdown-content thead th {
border-bottom: 2px solid var(--sl-color-accent);
text-align: left;
font-weight: 600;
letter-spacing: 0.02em;
text-transform: uppercase;
font-size: 0.78em;
color: var(--sl-color-text-accent);
}
.sl-markdown-content tbody td {
border-bottom: 1px solid var(--sl-color-hairline-light);
padding: 0.5rem 0.75rem;
}
/* Anchor links: underline, no rainbow */
.sl-markdown-content a:not(.sl-anchor-link) {
text-decoration: underline;
text-decoration-color: var(--sl-color-accent);
text-decoration-thickness: 1px;
text-underline-offset: 3px;
transition: text-decoration-thickness 0.1s ease;
}
.sl-markdown-content a:not(.sl-anchor-link):hover {
text-decoration-thickness: 2px;
}

5
docs-site/tsconfig.json Normal file
View File

@ -0,0 +1,5 @@
{
"extends": "astro/tsconfigs/strict",
"include": [".astro/types.d.ts", "**/*"],
"exclude": ["dist"]
}

View File

@ -1,6 +1,6 @@
[project] [project]
name = "informix-db" name = "informix-driver"
version = "2026.05.05.10" version = "2026.09.03"
description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries." description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries."
readme = "README.md" readme = "README.md"
license = { text = "MIT" } license = { text = "MIT" }
@ -27,9 +27,11 @@ classifiers = [
dependencies = [] dependencies = []
[project.urls] [project.urls]
Homepage = "https://github.com/rsp2k/informix-db" Homepage = "https://informix-driver.warehack.ing"
Documentation = "https://github.com/rsp2k/informix-db/tree/main/docs" Documentation = "https://informix-driver.warehack.ing"
Issues = "https://github.com/rsp2k/informix-db/issues" Source = "https://git.supported.systems/warehack.ing/informix-db"
Issues = "https://git.supported.systems/warehack.ing/informix-db/issues"
Changelog = "https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md"
[project.optional-dependencies] [project.optional-dependencies]
dev = [ dev = [
@ -49,13 +51,15 @@ packages = ["src/informix_db"]
# (the wheel doesn't ship these by default, but the sdist would). # (the wheel doesn't ship these by default, but the sdist would).
# See ~/.claude/rules/python.md for the full pre-publish PII audit playbook. # See ~/.claude/rules/python.md for the full pre-publish PII audit playbook.
exclude = [ exclude = [
"CLAUDE.md", # operator-private context "CLAUDE.md", # operator-private context
".env", ".env.local", ".env.*", ".env", ".env.local", ".env.*",
".mcp.json", # may contain local filesystem paths ".mcp.json", # may contain local filesystem paths
"build/", # decompiled JDBC, downloaded JARs "build/", # decompiled JDBC, downloaded JARs
"audits/", "audits/",
"docs/CAPTURES/", # spike artifacts; tests can re-capture against the dev container "docs/**", # protocol notes / decision log / captures — go to GitHub for the depth
"tests/reference/", # Java reference client — spike infra "docs-site/**", # Astro/Starlight site sources — published at informix-driver.warehack.ing
"tests/reference/**", # Java reference client — spike infra
"tests/benchmarks/.results/**", # pytest-benchmark cache (gitignored, but still ships unless excluded)
".pytest_cache/", ".ruff_cache/", ".mypy_cache/", ".pytest_cache/", ".ruff_cache/", ".mypy_cache/",
"dist/", "*.egg-info/", "dist/", "*.egg-info/",
] ]

View File

@ -23,6 +23,7 @@ from __future__ import annotations
import ssl import ssl
from importlib.metadata import PackageNotFoundError, version from importlib.metadata import PackageNotFoundError, version
from ._capabilities import ServerCapabilities
from .connections import Connection from .connections import Connection
from .converters import ( from .converters import (
BlobLocator, BlobLocator,
@ -49,6 +50,7 @@ from .pool import (
PoolTimeoutError, PoolTimeoutError,
create_pool, create_pool,
) )
from .rows import Row
# PEP 249 module-level globals # PEP 249 module-level globals
apilevel = "2.0" apilevel = "2.0"
@ -56,9 +58,14 @@ threadsafety = 1 # threads may share the module but not connections
paramstyle = "numeric" # locked in DECISION_LOG.md — matches Informix ESQL/C paramstyle = "numeric" # locked in DECISION_LOG.md — matches Informix ESQL/C
try: try:
__version__ = version("informix-db") # NOTE: the *distribution* is ``informix-driver``; the *module* is
# ``informix_db``. They differ, so this string can't be derived from
# ``__name__``. This looked up the old ``informix-db`` distribution
# until 2026.08.27, which meant __version__ silently reported
# "0.0.0+local" for anyone who installed the renamed package.
__version__ = version("informix-driver")
except PackageNotFoundError: except PackageNotFoundError:
# Editable install or running uninstalled; fall back to a sentinel. # Running from a source checkout without an install.
__version__ = "0.0.0+local" __version__ = "0.0.0+local"
__all__ = [ __all__ = [
@ -79,7 +86,9 @@ __all__ = [
"PoolClosedError", "PoolClosedError",
"PoolTimeoutError", "PoolTimeoutError",
"ProgrammingError", "ProgrammingError",
"Row",
"RowValue", "RowValue",
"ServerCapabilities",
"Warning", "Warning",
"__version__", "__version__",
"apilevel", "apilevel",
@ -104,6 +113,7 @@ def connect(
client_locale: str = "en_US.8859-1", client_locale: str = "en_US.8859-1",
env: dict[str, str] | None = None, env: dict[str, str] | None = None,
autocommit: bool = False, autocommit: bool = False,
row_factory: object | None = None,
tls: bool | ssl.SSLContext = False, tls: bool | ssl.SSLContext = False,
tls_server_hostname: str | None = None, tls_server_hostname: str | None = None,
) -> Connection: ) -> Connection:
@ -146,4 +156,5 @@ def connect(
client_locale=client_locale, client_locale=client_locale,
env=env, env=env,
autocommit=autocommit, autocommit=autocommit,
row_factory=row_factory,
) )

View File

@ -0,0 +1,269 @@
"""Server capability decoding — the SQ_PROTOCOLS feature bitmap.
SQLI does not have a single "protocol version". It has a 64-bit feature
bitmap that client and server negotiate at connect time, and several bits
in it change **wire framing**, not just feature availability. Get one of
those wrong and the row decoder desyncs.
We already perform the negotiation (``Connection._init_session`` sends
``SQ_PROTOCOLS`` with the same 8-byte client offer the IBM JDBC driver
uses). Until now we discarded the server's reply and hardcoded the modern
framing. This module decodes the reply so the assumptions can at least be
*checked* instead of merely believed.
Two independent sources feed the bitmap, mirroring
``IfxSqliConnect.getServerVer`` / ``enhancedProtocolMechanism``:
1. ``Cap_1`` from the login (SLTYPE_CONACC) response. This is the client's
own protocol level echoed back 316 means "speaks the enhanced
protocol". It is **not** a server version number, which is why JDBC
tests ``== 316`` rather than ``>=``.
2. The 8-byte mask from the SQ_PROTOCOLS reply.
When ``Cap_1`` is non-zero, JDBC unconditionally pre-sets bits
{0, 2, 3, 4, 49, 51} *before* applying the mask, and a Java ``BitSet.set``
only ever sets never clears. So those six bits are true on any modern
server regardless of what the mask says. ``isUSVER`` (bit 2) is one of
them, which is why it never varies in practice and is a dead end when
chasing version-specific behaviour.
Bit numbering is MSB-first within each byte: byte 0 bit 0x80 is bit 0,
byte 0 bit 0x01 is bit 7, byte 1 bit 0x80 is bit 8, and so on.
A note on length. The client offer is 8 bytes, but every server we have
tested replies with **nine**. JDBC's ``enhancedProtocolMechanism``
dispatches on ``switch (i) case 0..7`` and silently discards anything
past byte 7, so its ``BitSet(64)`` never sees the ninth byte. We decode
it bits 64-71 because it turns out to be the only part of the mask
that differs between releases::
Informix 15.0.1.0.3 bdbe9ffe7fb7ffef ff
Informix 14.10.FC7W1 bdbe9ffe7fb7ffef f8
Informix 12.10.FC12W1DE bdbe9ffe7fb7ffef f0
^^^^^^^^^^^^^^^^ identical
The first 64 bits being byte-identical across those three releases is
why they speak an indistinguishable SQLI dialect. Bits 64+ have no known
meaning here; they are surfaced for diagnostics, not branched on.
"""
from __future__ import annotations
from dataclasses import dataclass, field
# The client's capability offer. Byte-for-byte the value IBM's JDBC driver
# sends (``IfxSqliConnect.clientProtocols``), replayed verbatim — the bits
# are a fixed constant there too, never computed.
CLIENT_PROTOCOLS_MASK = bytes.fromhex("fffc7ffc3c8caa97")
# ``Cap_1`` value meaning "enhanced protocol negotiation applies". JDBC
# compares with equality, not >=, because the server echoes the client's
# own declared level rather than reporting its own.
ENHANCED_PROTOCOL_CAP = 316
# Bits JDBC pre-sets for any non-zero Cap_1, before the mask is applied.
_PRESET_BITS = frozenset({0, 2, 3, 4, 49, 51})
# Named bits. Only the ones we can justify from the decompiled JDBC are
# listed; the mask has 64 and most are irrelevant to us.
BIT_USVER = 2 # isUSVER — always true on modern servers
BIT_LONG_ID_A = 10 # isLongID is bit 10 OR bit 18
BIT_LONG_ID_B = 18
BIT_VARCHAR_VAR_LEN = 13 # gated further by bit 40 + client ifxPADVARCHAR
BIT_DESCRIBE_INPUT = 34
BIT_LVARCHAR_GT_2K = 37
BIT_PAD_VARCHAR_GATE = 40
BIT_FP_DESCRIBE = 45
BIT_AUTO_GENERATED_KEYS = 48
BIT_CUR_SESS_INFO = 49
BIT_FOUR_BYTE_OFFSET = 50 # describe: string-table + field-index widths
BIT_GLS = 51
BIT_NAMED_PARAMETERS = 52
BIT_BIGINT = 54
BIT_SAVEPOINT = 56
BIT_SQ_BATCH = 61
BIT_REMOVE_64K_LIMIT = 62 # 4-byte vs 2-byte length prefixes
BIT_TWO_GB_FETCH_BUFFER = 63
def _decode_bits(mask: bytes) -> set[int]:
"""Expand a big-endian, MSB-first bit mask into a set of bit numbers."""
bits: set[int] = set()
for byte_index, value in enumerate(mask):
for offset in range(8):
if value & (0x80 >> offset):
bits.add(byte_index * 8 + offset)
return bits
@dataclass(frozen=True)
class ServerCapabilities:
"""Decoded SQ_PROTOCOLS feature bitmap plus login-response metadata.
``bits`` is the authoritative set; the named properties are readability
sugar over it. ``raw_mask`` is kept so a bug report can include the
exact bytes without needing a packet capture.
"""
bits: frozenset[int] = field(default_factory=frozenset)
raw_mask: bytes = b""
cap_1: int = 0
cap_2: int = 0
cap_3: int = 0
server_version: str = ""
serial_number: str = ""
applid_name: str = ""
@classmethod
def from_wire(
cls,
mask: bytes,
*,
cap_1: int = 0,
cap_2: int = 0,
cap_3: int = 0,
server_version: str = "",
serial_number: str = "",
applid_name: str = "",
) -> ServerCapabilities:
bits = _decode_bits(mask)
if cap_1 != 0:
# Mirrors JDBC: pre-set bits are OR-ed in and never cleared.
bits |= _PRESET_BITS
return cls(
bits=frozenset(bits),
raw_mask=bytes(mask),
cap_1=cap_1,
cap_2=cap_2,
cap_3=cap_3,
server_version=server_version,
serial_number=serial_number,
applid_name=applid_name,
)
def has(self, bit: int) -> bool:
return bit in self.bits
# -- named capabilities ------------------------------------------------
@property
def usver(self) -> bool:
return self.has(BIT_USVER)
@property
def long_id(self) -> bool:
return self.has(BIT_LONG_ID_A) or self.has(BIT_LONG_ID_B)
@property
def varchar_var_len(self) -> bool:
"""VARCHAR carries a 1-byte length prefix rather than being padded
to its full declared width.
JDBC additionally consults the client-side ``ifxPADVARCHAR``
setting; we never enable padding, so the client half is constant.
"""
return self.has(BIT_VARCHAR_VAR_LEN)
@property
def four_byte_offset(self) -> bool:
"""Describe records use 4-byte string-table size and field indices
rather than 2-byte."""
return self.has(BIT_FOUR_BYTE_OFFSET)
@property
def bigint(self) -> bool:
return self.has(BIT_BIGINT)
@property
def lvarchar_gt_2k(self) -> bool:
return self.has(BIT_LVARCHAR_GT_2K)
@property
def remove_64k_limit(self) -> bool:
"""4-byte length prefixes in the fast-path/SQ_FILE paths."""
return self.has(BIT_REMOVE_64K_LIMIT)
@property
def two_gb_fetch_buffer(self) -> bool:
return self.has(BIT_TWO_GB_FETCH_BUFFER)
@property
def gls(self) -> bool:
return self.has(BIT_GLS)
@property
def describe_input(self) -> bool:
return self.has(BIT_DESCRIBE_INPUT)
@property
def named_parameters(self) -> bool:
return self.has(BIT_NAMED_PARAMETERS)
@property
def savepoint(self) -> bool:
return self.has(BIT_SAVEPOINT)
@property
def enhanced_protocol(self) -> bool:
return self.cap_1 == ENHANCED_PROTOCOL_CAP
# -- assumption checking ----------------------------------------------
def violated_assumptions(self) -> list[str]:
"""Framing assumptions this driver hardcodes that the server
contradicts.
Each entry names a place where we emit or parse a fixed wire shape
that is actually capability-gated. An empty list means every
hardcoded choice matches what the server negotiated.
This is a diagnostic, not a guarantee: it only covers bits we know
to be framing-relevant from the decompiled JDBC. It exists so that
a server we have never tested produces a loud, specific warning
instead of silently corrupted rows.
"""
problems: list[str] = []
if not self.four_byte_offset:
problems.append(
"server did not negotiate 4-byte describe offsets "
f"(bit {BIT_FOUR_BYTE_OFFSET}), but parse_describe reads "
"4-byte string-table size and field indices; column "
"metadata will be misparsed"
)
if not self.varchar_var_len:
problems.append(
f"server did not negotiate variable-length VARCHAR "
f"(bit {BIT_VARCHAR_VAR_LEN}); VARCHAR values are padded to "
"the declared column width on the wire, but the row decoder "
"expects a 1-byte length prefix"
)
if not self.remove_64k_limit:
problems.append(
f"server did not negotiate the 64K limit removal "
f"(bit {BIT_REMOVE_64K_LIMIT}); fast-path routine signatures "
"are emitted with a 4-byte length prefix that the server "
"expects to be 2-byte"
)
return problems
def __repr__(self) -> str:
named = [
name
for name, value in (
("usver", self.usver),
("4byte_offset", self.four_byte_offset),
("varchar_var_len", self.varchar_var_len),
("bigint", self.bigint),
("long_id", self.long_id),
("lvarchar>2k", self.lvarchar_gt_2k),
("remove_64k", self.remove_64k_limit),
("gls", self.gls),
)
if value
]
return (
f"ServerCapabilities(cap_1={self.cap_1}, "
f"mask={self.raw_mask.hex()}, "
f"version={self.server_version!r}, "
f"flags={'|'.join(named) or 'none'})"
)

View File

@ -39,9 +39,18 @@ def build_get_routine_pdu(signature: str) -> bytes:
``[short SQ_GETROUTINE=101][byte isRoutineById=0][int sigLen] ``[short SQ_GETROUTINE=101][byte isRoutineById=0][int sigLen]
[sig bytes][pad if odd][short fparamFlag=0][short SQ_EOT=12]`` [sig bytes][pad if odd][short fparamFlag=0][short SQ_EOT=12]``
JDBC's ``getJavaToIfxCharBytes`` uses 4-byte length prefix on JDBC's ``getJavaToIfxCharBytes`` uses a 4-byte length prefix when
modern servers (``isRemove64KLimitSupported``). We always emit the ``isRemove64KLimitSupported()`` (capability bit 62) is set, and a
4-byte form works against 12.10+ unequivocally. 2-byte prefix otherwise. We always emit the 4-byte form.
Verified correct on 15.0.1.0.3DE and 12.10.FC12W1DE (2026-05-08)
the fast-path RPC tests pass on both. Not yet checked on 14.10.
An earlier version of this docstring asserted 12.10+ compatibility
"unequivocally" before anyone had run it against 12.10. The claim
happened to hold, but it was a guess. The bit is negotiated via
``SQ_PROTOCOLS``, which this driver does not yet send, so this
remains an assumption on any server we haven't measured.
""" """
sig_bytes = signature.encode("iso-8859-1") sig_bytes = signature.encode("iso-8859-1")
sig_len = len(sig_bytes) sig_len = len(sig_bytes)

View File

@ -29,6 +29,29 @@ class ProtocolError(Exception):
"""Raised when wire bytes can't be parsed (truncated stream, bad framing).""" """Raised when wire bytes can't be parsed (truncated stream, bad framing)."""
# Every way a wire read/write can fail, as one tuple.
#
# This exists because the tuple was written out by hand in six places and
# drifted: four of them caught ``(ProtocolError, OSError)`` — which
# ``IfxSocket`` never raises. It converts *every* socket failure, including
# clean EOF, into ``OperationalError``, and that is a ``DatabaseError``, not
# an ``OSError``. So those four handlers could not catch the thing they
# existed to catch. The worst of them sat in a ``weakref.finalize`` callback,
# where the escaping exception is printed to stderr and swallowed, leaving a
# desynchronised connection to be returned to the pool marked healthy.
#
# Import this rather than re-typing the members. A tuple in one place cannot
# drift out of sync with itself.
def _wire_error_types() -> tuple[type[BaseException], ...]:
# Imported lazily: exceptions.py must stay free of protocol imports.
from .exceptions import InterfaceError, OperationalError
return (ProtocolError, OSError, OperationalError, InterfaceError)
WIRE_ERRORS: tuple[type[BaseException], ...] = _wire_error_types()
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Writer # Writer
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@ -209,6 +232,119 @@ class IfxStreamReader:
return raw.decode(encoding) return raw.decode(encoding)
class BufferedSocketReader(IfxStreamReader):
"""Phase 39: thin parser-view over the ``IfxSocket``'s persistent
read-ahead buffer.
The buffer itself (``_recv_buf`` + ``_recv_pos``) lives on the
``IfxSocket`` so it survives across reader instances. This is
critical for the pipelined-executemany path (Phase 33), which
streams N responses back-to-back a per-reader buffer would lose
pre-fetched bytes when one reader is destroyed and the next is
created.
Why this is faster than ``_SocketReader``:
* **One ``recv()`` per ~64 KB** instead of one per field. A 100k-row
SELECT calls ``read_exact`` ~450k times today; with this reader,
``recv`` fires ~hundreds of times instead.
* **No ``bytes.join``.** The bytearray IS the buffer; reads are
slices, not chunk-joins.
* **``struct.unpack_from(buf, offset)``** for fixed-width ints reads
directly from the bytearray at offset, avoiding the intermediate
slice the legacy reader created.
Each method reads through ``self._sock._recv_buf`` directly. The
``IfxSocket`` exposes ``fill_recv_buf(n)`` to ensure ``n`` bytes
are available from the current cursor position; this method
handles ``recv()``, compaction, and EOF detection.
``read_exact`` returns a fresh ``bytes`` (not a memoryview)
copying is cheap and decouples consumer lifetime from buffer
compaction, removing a class of use-after-compact bugs at
near-zero cost.
"""
__slots__ = ("_sock",)
def __init__(self, sock):
# Skip super().__init__ to avoid the unused BytesIO allocation
# — we override every method that touches ``self._source``.
self._sock = sock
def read_exact(self, n: int) -> bytes:
if n <= 0:
return b""
sock = self._sock
sock.fill_recv_buf(n)
buf = sock._recv_buf
pos = sock._recv_pos
end = pos + n
out = bytes(buf[pos:end])
sock._recv_pos = end
return out
def read_byte(self) -> int:
sock = self._sock
sock.fill_recv_buf(1)
b = sock._recv_buf[sock._recv_pos]
sock._recv_pos += 1
return b
def skip(self, n: int) -> None:
# The base reader's skip delegates to read_exact, which returns
# b"" for n <= 0. This one advances the cursor arithmetically, so
# without the guard a negative n *rewinds* it — silently
# re-decoding bytes already consumed as if they were the next
# field. Same divergence as read_exact, which does guard.
if n <= 0:
return
sock = self._sock
sock.fill_recv_buf(n)
sock._recv_pos += n
def read_padded(self, n: int) -> bytes:
out = self.read_exact(n)
if n & 1:
self.skip(1)
return out
def read_short(self) -> int:
sock = self._sock
sock.fill_recv_buf(2)
v = _FMT_SHORT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 2
return v
def read_int(self) -> int:
sock = self._sock
sock.fill_recv_buf(4)
v = _FMT_INT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 4
return v
def read_long_bigint(self) -> int:
sock = self._sock
sock.fill_recv_buf(8)
v = _FMT_LONG.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 8
return v
def read_real(self) -> float:
sock = self._sock
sock.fill_recv_buf(4)
v = _FMT_FLOAT.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 4
return v
def read_double(self) -> float:
sock = self._sock
sock.fill_recv_buf(8)
v = _FMT_DOUBLE.unpack_from(sock._recv_buf, sock._recv_pos)[0]
sock._recv_pos += 8
return v
def make_pdu_writer() -> tuple[IfxStreamWriter, BytesIO]: def make_pdu_writer() -> tuple[IfxStreamWriter, BytesIO]:
"""Convenience: create a writer backed by a fresh in-memory buffer. """Convenience: create a writer backed by a fresh in-memory buffer.

View File

@ -20,12 +20,21 @@ column names), read via readPadded.
from __future__ import annotations from __future__ import annotations
from collections.abc import Callable
from dataclasses import dataclass from dataclasses import dataclass
from datetime import timedelta as _timedelta
from types import MappingProxyType from types import MappingProxyType
from ._protocol import IfxStreamReader from ._protocol import IfxStreamReader, ProtocolError
from ._types import IfxType, base_type, is_nullable from ._types import IfxType, base_type, is_nullable
from .converters import ( from .converters import (
_DOUBLE_NULL,
_REAL_NULL,
_UNPACK_DOUBLE,
_UNPACK_FLOAT,
_UNPACK_INT,
_UNPACK_LONG,
_UNPACK_SHORT,
DECODERS, DECODERS,
FIXED_WIDTHS, FIXED_WIDTHS,
BlobLocator, BlobLocator,
@ -36,6 +45,9 @@ from .converters import (
_decode_datetime, _decode_datetime,
_decode_interval, _decode_interval,
) )
from .converters import (
_INFORMIX_DATE_EPOCH as _DATE_EPOCH,
)
# Module-level type-code constants — lifted out of the hot loop in # Module-level type-code constants — lifted out of the hot loop in
# parse_tuple_payload so we don't pay the IntFlag→int conversion per # parse_tuple_payload so we don't pay the IntFlag→int conversion per
@ -214,6 +226,17 @@ _LENGTH_PREFIXED_SHORT_TYPES = frozenset({
_TC_NVCHAR, _TC_NVCHAR,
}) })
# CHAR and NCHAR are **fixed-width**, space-padded to ``encoded_length``.
# VARCHAR and NVARCHAR are byte-length-prefixed. Getting NCHAR wrong is not
# a cosmetic bug: treating it as length-prefixed consumes its first
# character as a length byte, then advances the offset by that value —
# e.g. NCHAR(10) holding 'nch' reads 0x6E ('n') as a 110-byte length and
# desyncs the rest of the row, usually into a struct.error crash.
# Verified on the wire against 12.10 and 15:
# NCHAR(10) 'nch' -> 6e 63 68 20 20 20 20 20 20 20 (10 B, padded)
# NVARCHAR(20) 'nvc' -> 03 6e 76 63 (1-byte prefix)
_FIXED_WIDTH_CHAR_TYPES = frozenset({_TC_CHAR, _TC_NCHAR})
_COMPOSITE_UDT_TYPES = frozenset({ _COMPOSITE_UDT_TYPES = frozenset({
_TC_ROW, _TC_ROW,
_TC_COLLECTION, _TC_COLLECTION,
@ -224,6 +247,140 @@ _COMPOSITE_UDT_TYPES = frozenset({
_NUMERIC_TYPES = frozenset({_TC_DECIMAL, _TC_MONEY}) _NUMERIC_TYPES = frozenset({_TC_DECIMAL, _TC_MONEY})
def _read_udt_envelope(payload: bytes, offset: int) -> tuple[int, bytes | None]:
"""Read the UDT wire envelope: ``[1-byte indicator][int32 length][data]``.
Returns ``(new_offset, body)`` with ``body`` ``None`` when the
indicator says NULL. The length field is present either way it
belongs to the envelope, not the value so the offset advances by
5 for a NULL and 5 + length otherwise.
This exists because the same envelope was hand-written four times and
three of the copies were wrong, each in a different way:
* BOOLEAN read ``encoded_length`` (1 byte) instead of the envelope,
leaving 5 bytes behind.
* The composite-UDT branch returned on the indicator without reading
the length, leaving 4 bytes behind.
* BLOB/CLOB read a flat 72 bytes, leaving 77 behind.
Only the UDTVAR(lvarchar) copy was right, and only after two separate
fixes. Four copies of one three-line format is four chances to get it
wrong; this is one.
"""
indicator = payload[offset]
offset += 1
length = int.from_bytes(payload[offset : offset + 4], "big", signed=True)
offset += 4
if indicator == 1:
return offset, None
if length < 0:
raise ProtocolError(
f"UDT envelope declared a negative length ({length}) at payload "
f"offset {offset - 4}; the wire is desynchronised"
)
body = payload[offset : offset + length]
offset += length
return offset, body
def _decode_lob_locator(body: bytes, extended_id: int):
"""Turn a smart-LOB envelope body into a Blob/ClobLocator.
The body is the 72-byte locator **hex-encoded as 144 ASCII characters**,
not the raw locator. We previously consumed a flat 72 bytes, which took
the first half of that hex text and handed it to ``BlobLocator`` as if
it were binary so ``BlobLocator.raw`` never actually held a locator.
It went unnoticed because ``read_blob_column`` resolves LOBs through
the server-side ``lotofile`` function and never uses the locator.
"""
cls = BlobLocator if extended_id == 10 else ClobLocator
try:
raw = bytes.fromhex(body.decode("ascii"))
except (ValueError, UnicodeDecodeError):
# Not hex — surface what arrived rather than guessing, but only if
# it is already the right size for a locator.
raw = bytes(body)
if len(raw) != 72:
raise ProtocolError(
f"smart-LOB locator decoded to {len(raw)} bytes, expected 72 "
f"(envelope body was {len(body)} bytes)"
)
return cls(raw=raw)
def _row_not_consumed(
offset: int, payload_len: int, columns: list[ColumnInfo]
) -> None:
"""Raise when a decoded row didn't consume exactly its payload.
THIS IS THE CHECK THAT WOULD HAVE CAUGHT EVERY FRAMING BUG WE SHIPPED.
Thirteen of them reached users, and all thirteen were the same defect:
a column consumed the wrong number of bytes. Not one raised at the
point of the mistake the wrong width produced a plausible value and
corrupted whatever came *next*, so the damage surfaced in a different
column, a different row, or a different statement.
Every one was detectable here for free. ``payload`` is a fully
extracted ``bytes`` object of known length (the wire's even-alignment
pad is consumed separately by the caller), so after decoding N columns
the offset must land exactly on the end. Anything else means the codec
and the server disagree about the row's shape, and continuing can only
produce wrong answers.
A previous attempt at this check was reverted, and the comment
explaining why survived long after its reason did: it cited the
LVARCHAR odd-length pad, which was itself a bug and has since been
removed. With that gone the invariant is exact.
The message names the column shape because "row decode failed" is not
actionable the whole point is to say *which* column shape and by how
many bytes, so the next report arrives with the answer in it.
"""
delta = offset - payload_len
direction = "over-read" if delta > 0 else "under-read"
shape = ", ".join(
f"{c.name}:tc={c.type_code}"
f"{'/ext=' + c.extended_name if c.extended_name else ''}"
f"/enclen={c.encoded_length}"
for c in columns
)
raise ProtocolError(
f"row decoder {direction} by {abs(delta)} byte(s): consumed "
f"{offset} of {payload_len} payload bytes. This means the driver "
f"and the server disagree about a column's wire width, so every "
f"column after the offending one is unreliable. Column shape: "
f"[{shape}]"
)
def _packed_width(encoded_length: int) -> int:
"""On-wire byte width for the four types whose ``encoded_length``
packs two fields into ``(high << 8) | low``: DECIMAL, MONEY,
DATETIME and INTERVAL.
Mirrors ``IfxColumnInfo.adjustedColumnLength`` / ``IfxDecimal.decLength``::
((ColLength >> 8 & 0xFF) + (ColLength & 0xFF & 1) + 3) / 2
The ``low & 1`` term is the part that is easy to miss, and dropping
it is wrong exactly when the high byte is even and the low byte is
odd. For DECIMAL/MONEY the low byte is the scale, and an unscaled
``DECIMAL(p)`` reports scale **255** odd so every floating
DECIMAL with even precision was read one byte short, desyncing every
column after it. ``DECIMAL(16)`` is 10 bytes on the wire, not 9.
For DATETIME and INTERVAL the low byte is the qualifier, whose
parity happens to track the digit count, so the two formulas agree
there in practice. They are unified here anyway: one rule taken from
the reference beats two that coincide for reasons nobody wrote down.
"""
high = (encoded_length >> 8) & 0xFF
low = encoded_length & 0xFF
return (high + (low & 1) + 3) // 2
# Types that are fixed-width on the wire AND have a registered decoder # Types that are fixed-width on the wire AND have a registered decoder
# in ``FIXED_WIDTHS``: SMALLINT, INT, SERIAL, SMFLOAT, FLOAT, BIGINT, # in ``FIXED_WIDTHS``: SMALLINT, INT, SERIAL, SMFLOAT, FLOAT, BIGINT,
# BIGSERIAL, DATE, BOOL. These are the most common types in any real # BIGSERIAL, DATE, BOOL. These are the most common types in any real
@ -281,12 +438,13 @@ def compile_column_readers(columns: list[ColumnInfo]) -> list[tuple]:
readers.append((_RK_FIXED, FIXED_WIDTHS[tc], DECODERS[tc])) readers.append((_RK_FIXED, FIXED_WIDTHS[tc], DECODERS[tc]))
continue continue
if tc == _TC_CHAR: if tc in _FIXED_WIDTH_CHAR_TYPES:
# CHAR and NCHAR: fixed width, space-padded to encoded_length.
readers.append((_RK_CHAR, col.encoded_length, DECODERS[tc])) readers.append((_RK_CHAR, col.encoded_length, DECODERS[tc]))
continue continue
if tc in _LENGTH_PREFIXED_SHORT_TYPES: if tc in _LENGTH_PREFIXED_SHORT_TYPES:
# VARCHAR / NCHAR / NVCHAR — CHAR was already excluded above. # VARCHAR / NVARCHAR — CHAR and NCHAR excluded above.
readers.append((_RK_BYTE_PREFIX, DECODERS[tc])) readers.append((_RK_BYTE_PREFIX, DECODERS[tc]))
continue continue
@ -295,20 +453,17 @@ def compile_column_readers(columns: list[ColumnInfo]) -> list[tuple]:
continue continue
if tc in _NUMERIC_TYPES: if tc in _NUMERIC_TYPES:
precision = (col.encoded_length >> 8) & 0xFF width = _packed_width(col.encoded_length)
width = (precision + 1) // 2 + 1
readers.append((_RK_DECIMAL, width, DECODERS[tc])) readers.append((_RK_DECIMAL, width, DECODERS[tc]))
continue continue
if tc == _TC_DATETIME: if tc == _TC_DATETIME:
digit_count = (col.encoded_length >> 8) & 0xFF width = _packed_width(col.encoded_length)
width = (digit_count + 1) // 2 + 1
readers.append((_RK_DATETIME, width, col.encoded_length)) readers.append((_RK_DATETIME, width, col.encoded_length))
continue continue
if tc == _TC_INTERVAL: if tc == _TC_INTERVAL:
digit_count = (col.encoded_length >> 8) & 0xFF width = _packed_width(col.encoded_length)
width = (digit_count + 1) // 2 + 1
readers.append((_RK_INTERVAL, width, col.encoded_length)) readers.append((_RK_INTERVAL, width, col.encoded_length))
continue continue
@ -318,6 +473,248 @@ def compile_column_readers(columns: list[ColumnInfo]) -> list[tuple]:
return readers return readers
# Phase 38 codegen — sentinel constants imported into the generated
# function's globals so inlined decode bodies can reference them by
# name without dotted lookups.
_INT_MIN_SENTINEL = -0x80000000
_SHORT_MIN_SENTINEL = -0x8000
_LONG_MIN_SENTINEL = -0x8000000000000000
def compile_row_decoder(
readers: list[tuple],
columns: list[ColumnInfo],
) -> Callable[[bytes, int, str], tuple] | None:
"""Generate a specialized row decoder for a specific column shape.
Phase 38: takes the Phase 37 reader-list and emits a Python
function via ``exec()`` that decodes one row of this exact shape
in straight-line code no per-column iteration, no per-column
tuple-unpack, no per-column branch dispatch. Each column's
decode logic is inlined directly.
The generated function has signature
``parse_row(payload, offset, encoding) -> tuple`` and only
references module-level helpers via its closure-equivalent
globals dict (the ``_g`` dict below).
Returns ``None`` if any column's reader-kind is unsupported by
the codegen caller falls back to the Phase 37 dispatch loop.
The generated source is printable via ``IFX_DEBUG_CODEGEN=1``
env var for inspection / debugging.
"""
import os
lines: list[str] = []
lines.append("def parse_row(payload, offset, encoding):")
val_names: list[str] = []
# Map type-code → inline-decoder source for the common fixed-width
# decoders. Inlining the decoder body eliminates one function call
# per column — the actual codegen win. For types not in this map,
# fall back to ``_D{i}(raw)`` referencing the decoder via globals.
_INLINE_FIXED = {
# type_code: lambda v, raw_var: source-snippet
# SMALLINT (1)
1: lambda v, r: (
f" {v} = _UNPACK_SHORT({r})[0]\n"
f" if {v} == -32768:\n"
f" {v} = None"
),
# INT (2), SERIAL (6) — same body
2: lambda v, r: (
f" {v} = _UNPACK_INT({r})[0]\n"
f" if {v} == -2147483648:\n"
f" {v} = None"
),
6: lambda v, r: (
f" {v} = _UNPACK_INT({r})[0]\n"
f" if {v} == -2147483648:\n"
f" {v} = None"
),
# BIGINT (52), BIGSERIAL (53) — same body
52: lambda v, r: (
f" {v} = _UNPACK_LONG({r})[0]\n"
f" if {v} == -9223372036854775808:\n"
f" {v} = None"
),
53: lambda v, r: (
f" {v} = _UNPACK_LONG({r})[0]\n"
f" if {v} == -9223372036854775808:\n"
f" {v} = None"
),
# FLOAT (3), SMFLOAT (4)
3: lambda v, r: (
f" if {r} == _DOUBLE_NULL:\n"
f" {v} = None\n"
f" else:\n"
f" {v} = _UNPACK_DOUBLE({r})[0]"
),
4: lambda v, r: (
f" if {r} == _REAL_NULL:\n"
f" {v} = None\n"
f" else:\n"
f" {v} = _UNPACK_FLOAT({r})[0]"
),
# DATE (7) — 4-byte day count from 1899-12-31
7: lambda v, r: (
f" days = _UNPACK_INT({r})[0]\n"
f" if days == -2147483648:\n"
f" {v} = None\n"
f" else:\n"
f" {v} = _DATE_EPOCH + _timedelta(days=days)"
),
# BOOL (45) — left to the canonical decoder. Informix BOOL is
# ``'t'/'T'/1``, NOT bool(byte) — a truthy-byte inline would
# silently turn ``'f'`` (102) into True.
}
for i, r in enumerate(readers):
kind = r[0]
v = f"v{i}"
val_names.append(v)
lines.append(f" # Col {i}: kind={kind}")
if kind == _RK_FIXED:
_, width, _decoder = r
lines.append(f" raw = payload[offset:offset+{width}]")
lines.append(f" offset += {width}")
# Find type code from the decoder identity (we don't have
# tc directly in the reader tuple; recover via the columns
# list).
tc = columns[i].type_code
inline_src = _INLINE_FIXED.get(tc)
if inline_src is not None:
lines.append(inline_src(v, "raw"))
else:
lines.append(f" {v} = _D{i}(raw)")
elif kind == _RK_BYTE_PREFIX:
lines.append(" length = payload[offset]")
lines.append(" offset += 1")
lines.append(" raw = payload[offset:offset + length]")
lines.append(" offset += length")
lines.append(f" {v} = _D{i}(raw, encoding)")
elif kind == _RK_CHAR:
_, width, _decoder = r
lines.append(f" raw = payload[offset:offset+{width}]")
lines.append(f" offset += {width}")
lines.append(f" {v} = _D{i}(raw, encoding)")
elif kind == _RK_LVARCHAR:
lines.append(
" length = int.from_bytes("
"payload[offset:offset+4], 'big', signed=True)"
)
lines.append(" offset += 4")
lines.append(" raw = payload[offset:offset + length]")
lines.append(" offset += length")
# No even-byte pad — see _TC_LVARCHAR in the legacy chain.
lines.append(f" {v} = _D{i}(raw, encoding)")
elif kind == _RK_DECIMAL:
_, width, _decoder = r
lines.append(f" raw = payload[offset:offset+{width}]")
lines.append(f" offset += {width}")
lines.append(" try:")
lines.append(f" {v} = _D{i}(raw)")
lines.append(" except NotImplementedError:")
lines.append(f" {v} = raw")
elif kind == _RK_DATETIME:
_, width, enc_len = r
lines.append(f" raw = payload[offset:offset+{width}]")
lines.append(f" offset += {width}")
lines.append(f" {v} = _decode_datetime(raw, {enc_len})")
elif kind == _RK_INTERVAL:
_, width, enc_len = r
lines.append(f" raw = payload[offset:offset+{width}]")
lines.append(f" offset += {width}")
lines.append(f" {v} = _decode_interval(raw, {enc_len})")
elif kind == _RK_LEGACY:
# Codegen for rare types: call the legacy helper. The
# column metadata is referenced via the globals dict.
tc = r[1]
lines.append(
f" offset, {v} = _legacy_dispatch_one_column("
f"payload, offset, {tc}, _COL{i}, encoding)"
)
else:
# Unknown kind — abort codegen, caller falls back.
return None
# Same end-of-row reconciliation the interpreted paths do. The
# generated function is the hot path, so this must be emitted here
# too — a check that only guards the slow path guards nothing in
# production.
lines.append(" if offset != len(payload):")
lines.append(" _row_short(offset, len(payload))")
if val_names:
lines.append(f" return ({', '.join(val_names)},)")
else:
lines.append(" return ()")
src = "\n".join(lines)
if os.environ.get("IFX_DEBUG_CODEGEN") == "1":
import sys
print("=== informix_db codegen ===", file=sys.stderr)
print(src, file=sys.stderr)
print("=== end ===", file=sys.stderr)
# Build the globals dict for the generated function. Each column's
# decoder (if any) is registered as ``_D<i>``; columns with the
# _RK_LEGACY kind get their ColumnInfo as ``_COL<i>``.
#
# The inlined fixed-width snippets (see ``_INLINE_FIXED`` above)
# reference precompiled struct unpackers and NULL sentinels by
# name — they only resolve if we hand them to ``exec`` here.
g: dict = {
"_decode_datetime": _decode_datetime,
"_decode_interval": _decode_interval,
"_legacy_dispatch_one_column": _legacy_dispatch_one_column,
# Bound to this shape's columns so the raised message can name them
# without the generated source having to carry the list itself.
"_row_short": (
lambda offset, payload_len, _cols=columns: _row_not_consumed(
offset, payload_len, _cols
)
),
"_UNPACK_SHORT": _UNPACK_SHORT,
"_UNPACK_INT": _UNPACK_INT,
"_UNPACK_LONG": _UNPACK_LONG,
"_UNPACK_FLOAT": _UNPACK_FLOAT,
"_UNPACK_DOUBLE": _UNPACK_DOUBLE,
"_DOUBLE_NULL": _DOUBLE_NULL,
"_REAL_NULL": _REAL_NULL,
"_DATE_EPOCH": _DATE_EPOCH,
"_timedelta": _timedelta,
"int": int, # ensure the builtin isn't shadowed
"bool": bool,
}
for i, r in enumerate(readers):
kind = r[0]
if kind in (_RK_FIXED, _RK_CHAR, _RK_DECIMAL):
g[f"_D{i}"] = r[2]
elif kind in (_RK_BYTE_PREFIX, _RK_LVARCHAR):
g[f"_D{i}"] = r[1]
elif kind == _RK_LEGACY:
g[f"_COL{i}"] = columns[i]
namespace: dict = {}
try:
exec(compile(src, "<informix_db codegen>", "exec"), g, namespace)
except SyntaxError:
return None
return namespace["parse_row"]
def _legacy_dispatch_one_column( def _legacy_dispatch_one_column(
payload: bytes, payload: bytes,
offset: int, offset: int,
@ -334,20 +731,51 @@ def _legacy_dispatch_one_column(
""" """
# BLOB / CLOB locator (UDTFIXED + extended_id 10/11) # BLOB / CLOB locator (UDTFIXED + extended_id 10/11)
if tc == _TC_UDTFIXED and col.extended_id in (10, 11): if tc == _TC_UDTFIXED and col.extended_id in (10, 11):
width = col.encoded_length # Smart LOBs use the UDT envelope, NOT a flat encoded_length field.
raw = payload[offset:offset + width] # Measured on Informix 15: a populated BLOB column is 149 bytes
offset += width # ([ind=0][len=144][144 hex chars]) and a NULL one is 5, while
cls = BlobLocator if col.extended_id == 10 else ClobLocator # encoded_length reports 72. Consuming 72 left 77 bytes behind and
return offset, cls(raw=bytes(raw)) # corrupted every column after a LOB. Found by the end-of-row
# reconciliation check, which fired on its first run.
offset, body = _read_udt_envelope(payload, offset)
if body is None:
return offset, None
return offset, _decode_lob_locator(body, col.extended_id)
# BOOLEAN. The server describes it as UDTFIXED (41) with
# extended_name='boolean' and encoded_length=1, but ``encoded_length``
# is the size of the *value*, not the field: on the wire it carries the
# standard UDT envelope ``[1-byte null indicator][4-byte length][data]``
# — 6 bytes total for a 1-byte value. Consuming only ``encoded_length``
# leaves 5 bytes on the wire and desyncs every subsequent column.
# Verified payload (INT, BOOLEAN 't', INT, VARCHAR 'tail'):
# 00 01 b2 07 | 00 00 00 00 01 74 | 00 03 64 0e | 04 74 61 69 6c
# The value byte is 0x74 ('t'), which _decode_bool already understands.
if tc == _TC_UDTFIXED and (
col.extended_name == "boolean" or col.extended_id == 5
):
offset, body = _read_udt_envelope(payload, offset)
if body is None:
return offset, None
return offset, bool(body and body[0] in (ord("t"), ord("T"), 1))
# ROW / COLLECTION composite UDT # ROW / COLLECTION composite UDT
if tc in _COMPOSITE_UDT_TYPES: if tc in _COMPOSITE_UDT_TYPES:
indicator = payload[offset] indicator = payload[offset]
offset += 1 offset += 1
if indicator == 1: # The 4-byte length is part of the envelope and is present even when
return offset, None # the indicator says NULL — identical to the UDTVAR(lvarchar) branch
# below, which decodes the same `[ind][int32 len][data]` shape.
# Returning early on the indicator left those 4 bytes unread and
# corrupted the following column. Wire evidence, Informix 15,
# (INT, SET(INT), VARCHAR) with a NULL set:
# 00 00 00 08 | 01 | 00 00 00 00 | 03 78 79 7a
# a = 8 | ind | length = 0 | [3]"xyz"
# Before the fix the VARCHAR decoded as '' instead of 'xyz'.
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True) length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
offset += 4 offset += 4
if indicator == 1:
return offset, None
raw = bytes(payload[offset:offset + length]) raw = bytes(payload[offset:offset + length])
offset += length offset += length
if tc == _TC_ROW: if tc == _TC_ROW:
@ -362,14 +790,35 @@ def _legacy_dispatch_one_column(
if tc == _TC_UDTVAR and col.extended_name == "lvarchar": if tc == _TC_UDTVAR and col.extended_name == "lvarchar":
indicator = payload[offset] indicator = payload[offset]
offset += 1 offset += 1
if indicator == 1: # The 4-byte length is present even when the indicator says NULL —
return offset, None # it is part of the envelope, not of the value. Returning early on
# the indicator left those 4 bytes on the wire and desynced every
# following column. Wire evidence (Informix 12.10), INT8 / LVARCHAR
# / INT8 with the middle column varying:
# 'odd' -> 00 | 00 00 00 03 | 6f 64 64 (8 bytes)
# NULL -> 01 | 00 00 00 00 (5 bytes)
# '' -> 00 | 00 00 00 00 (5 bytes)
# Note NULL and empty-string differ only in the indicator byte.
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True) length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
offset += 4 offset += 4
if indicator == 1:
return offset, None
raw = payload[offset:offset + length] raw = payload[offset:offset + length]
offset += length offset += length
if length & 1: # NO even-byte pad. The UDT envelope is [indicator][int len][bytes]
offset += 1 # and the next column starts immediately after the last content
# byte. Verified on the wire (Informix 12.10) for an odd-length
# value — INT8(2001), LVARCHAR('PackageRoot'), INT8(10):
# 00 01 00 00 07 d1 00 00 00 00 a = INT8 2001
# 00 null indicator
# 00 00 00 0b length 11
# 50 61 63 6b 61 67 65 52 6f 6f 74 "PackageRoot" (odd)
# 00 01 00 00 00 0a 00 00 00 00 b = INT8 10, starts immediately
# A pad here consumed one byte too many and shifted every
# subsequent column: INT8 10 decoded as 2560 (0x0A00), strings
# lost their first character, and wide rows ran off the end of
# the payload. Only fired for odd-length values, which is why the
# test fixture ("lv value", 8 chars) never caught it.
return offset, raw.decode(encoding) return offset, raw.decode(encoding)
# Unknown — surface ``encoded_length`` bytes raw. # Unknown — surface ``encoded_length`` bytes raw.
@ -387,6 +836,7 @@ def parse_tuple_payload(
columns: list[ColumnInfo], columns: list[ColumnInfo],
encoding: str = "iso-8859-1", encoding: str = "iso-8859-1",
readers: list[tuple] | None = None, readers: list[tuple] | None = None,
row_decoder: Callable[[bytes, int, str], tuple] | None = None,
) -> tuple: ) -> tuple:
"""Parse a SQ_TUPLE payload (the SQ_TUPLE tag is already consumed). """Parse a SQ_TUPLE payload (the SQ_TUPLE tag is already consumed).
@ -419,6 +869,13 @@ def parse_tuple_payload(
if size & 1: if size & 1:
reader.read_exact(1) reader.read_exact(1)
# Phase 38 fastest path: a per-result-set decoder function compiled
# via ``exec()`` from the column shape (see ``compile_row_decoder``).
# All per-column dispatch is eliminated — each column's decode logic
# is inlined in straight-line code.
if row_decoder is not None:
return row_decoder(payload, 0, encoding)
values: list[object] = [] values: list[object] = []
offset = 0 offset = 0
@ -463,8 +920,7 @@ def parse_tuple_payload(
offset += 4 offset += 4
raw = payload[offset:offset + length] raw = payload[offset:offset + length]
offset += length offset += length
if length & 1: # No even-byte pad — see _TC_LVARCHAR in the legacy chain.
offset += 1
values.append(decoder(raw, encoding)) values.append(decoder(raw, encoding))
continue continue
@ -504,6 +960,8 @@ def parse_tuple_payload(
payload, offset, tc, col, encoding payload, offset, tc, col, encoding
) )
values.append(value) values.append(value)
if offset != len(payload):
_row_not_consumed(offset, len(payload), columns)
return tuple(values) return tuple(values)
# Legacy slow path (no pre-compiled readers). # Legacy slow path (no pre-compiled readers).
@ -535,8 +993,8 @@ def parse_tuple_payload(
# docs/CAPTURES/13-py-varchar.socat.log: # docs/CAPTURES/13-py-varchar.socat.log:
# payload = 09 73 79 73 74 61 62 6c 65 73 # payload = 09 73 79 73 74 61 62 6c 65 73
# = [byte 9]["systables"] # = [byte 9]["systables"]
# CHAR is fixed-width per encoded_length — handled below. # CHAR and NCHAR are fixed-width per encoded_length.
if tc == _TC_CHAR: if tc in _FIXED_WIDTH_CHAR_TYPES:
width = col.encoded_length width = col.encoded_length
raw = payload[offset:offset + width] raw = payload[offset:offset + width]
offset += width offset += width
@ -548,14 +1006,22 @@ def parse_tuple_payload(
values.append(_decode_base(tc, raw, encoding)) values.append(_decode_base(tc, raw, encoding))
continue continue
# LVARCHAR as a bare type code (43), i.e. without the UDT envelope:
# ``[int length][bytes]``, no even-byte pad.
#
# Caveat worth stating plainly: every Informix server we have tested
# (12.10, 14.10, 15) describes *all* LVARCHAR columns as UDTVAR (40)
# with extended_name='lvarchar' — including casts like
# ``'abc'::LVARCHAR`` — so this branch is unreachable in practice and
# its framing is inferred from the UDTVAR evidence rather than
# observed directly. It is kept consistent with that branch on the
# reasoning that the content encoding shouldn't depend on how the
# column happens to be described.
if tc == _TC_LVARCHAR: if tc == _TC_LVARCHAR:
# [int length][bytes][pad if odd]
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True) length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
offset += 4 offset += 4
raw = payload[offset:offset + length] raw = payload[offset:offset + length]
offset += length offset += length
if length & 1:
offset += 1
values.append(_decode_base(tc, raw, encoding)) values.append(_decode_base(tc, raw, encoding))
continue continue
@ -563,8 +1029,7 @@ def parse_tuple_payload(
# the high byte of encoded_length (packed as (precision << 8) | scale). # the high byte of encoded_length (packed as (precision << 8) | scale).
# Per IfxRowColumn.loadColumnData and IfxToJavaDecimal byte sizing. # Per IfxRowColumn.loadColumnData and IfxToJavaDecimal byte sizing.
if tc in _NUMERIC_TYPES: if tc in _NUMERIC_TYPES:
precision = (col.encoded_length >> 8) & 0xFF width = _packed_width(col.encoded_length)
width = (precision + 1) // 2 + 1
raw = payload[offset:offset + width] raw = payload[offset:offset + width]
offset += width offset += width
try: try:
@ -578,8 +1043,7 @@ def parse_tuple_payload(
# (start_TU << 4) | end_TU). The decoder needs the qualifier too, # (start_TU << 4) | end_TU). The decoder needs the qualifier too,
# so we call it directly here rather than via the dispatch. # so we call it directly here rather than via the dispatch.
if tc == _TC_DATETIME: if tc == _TC_DATETIME:
digit_count = (col.encoded_length >> 8) & 0xFF width = _packed_width(col.encoded_length)
width = (digit_count + 1) // 2 + 1
raw = payload[offset:offset + width] raw = payload[offset:offset + width]
offset += width offset += width
values.append(_decode_datetime(raw, col.encoded_length)) values.append(_decode_datetime(raw, col.encoded_length))
@ -592,8 +1056,7 @@ def parse_tuple_payload(
# qualifier is needed at decode time, so we bypass the generic # qualifier is needed at decode time, so we bypass the generic
# dispatch. # dispatch.
if tc == _TC_INTERVAL: if tc == _TC_INTERVAL:
digit_count = (col.encoded_length >> 8) & 0xFF width = _packed_width(col.encoded_length)
width = (digit_count + 1) // 2 + 1
raw = payload[offset:offset + width] raw = payload[offset:offset + width]
offset += width offset += width
values.append(_decode_interval(raw, col.encoded_length)) values.append(_decode_interval(raw, col.encoded_length))
@ -605,11 +1068,25 @@ def parse_tuple_payload(
# we read here are an opaque server-side reference, NOT the # we read here are an opaque server-side reference, NOT the
# actual data. Phase 10 lets users fetch via lotofile + SQ_FILE. # actual data. Phase 10 lets users fetch via lotofile + SQ_FILE.
if tc == _TC_UDTFIXED and col.extended_id in (10, 11): if tc == _TC_UDTFIXED and col.extended_id in (10, 11):
width = col.encoded_length # UDT envelope, not a flat encoded_length field — see the
raw = payload[offset:offset + width] # matching branch in _legacy_dispatch_one_column.
offset += width offset, body = _read_udt_envelope(payload, offset)
cls = BlobLocator if col.extended_id == 10 else ClobLocator values.append(
values.append(cls(raw=bytes(raw))) None if body is None
else _decode_lob_locator(body, col.extended_id)
)
continue
# BOOLEAN — UDT envelope, not a bare byte. See the matching branch
# in _legacy_dispatch_one_column for the wire evidence.
if tc == _TC_UDTFIXED and (
col.extended_name == "boolean" or col.extended_id == 5
):
offset, body = _read_udt_envelope(payload, offset)
values.append(
None if body is None
else bool(body and body[0] in (ord("t"), ord("T"), 1))
)
continue continue
# ROW / COLLECTION (Phase 12): composite UDTs. Wire format is # ROW / COLLECTION (Phase 12): composite UDTs. Wire format is
@ -626,13 +1103,15 @@ def parse_tuple_payload(
if tc in _COMPOSITE_UDT_TYPES: if tc in _COMPOSITE_UDT_TYPES:
indicator = payload[offset] indicator = payload[offset]
offset += 1 offset += 1
if indicator == 1: # null # Length is present even when NULL — see the matching branch in
values.append(None) # _legacy_dispatch_one_column for the wire evidence.
continue
length = int.from_bytes( length = int.from_bytes(
payload[offset:offset + 4], "big", signed=True payload[offset:offset + 4], "big", signed=True
) )
offset += 4 offset += 4
if indicator == 1: # null
values.append(None)
continue
raw = bytes(payload[offset:offset + length]) raw = bytes(payload[offset:offset + length])
offset += length offset += length
if tc == _TC_ROW: if tc == _TC_ROW:
@ -657,17 +1136,18 @@ def parse_tuple_payload(
if tc == _TC_UDTVAR and col.extended_name == "lvarchar": if tc == _TC_UDTVAR and col.extended_name == "lvarchar":
indicator = payload[offset] indicator = payload[offset]
offset += 1 offset += 1
if indicator == 1: # Length is present even when NULL, and there is no even-byte
values.append(None) # pad — see the matching branch in _legacy_dispatch_one_column
continue # for the wire evidence.
length = int.from_bytes( length = int.from_bytes(
payload[offset:offset + 4], "big", signed=True payload[offset:offset + 4], "big", signed=True
) )
offset += 4 offset += 4
if indicator == 1:
values.append(None)
continue
raw = payload[offset:offset + length] raw = payload[offset:offset + length]
offset += length offset += length
if length & 1:
offset += 1
values.append(raw.decode(encoding)) values.append(raw.decode(encoding))
continue continue
@ -697,4 +1177,6 @@ def parse_tuple_payload(
# by Python's slicing semantics for strings — short = harmless). # by Python's slicing semantics for strings — short = harmless).
# If a future protocol message produces actual garbage here, add a # If a future protocol message produces actual garbage here, add a
# branch-local check at the offending dispatch path. # branch-local check at the offending dispatch path.
if offset != len(payload):
_row_not_consumed(offset, len(payload), columns)
return tuple(values) return tuple(values)

View File

@ -17,11 +17,35 @@ the rest of the protocol layer.
from __future__ import annotations from __future__ import annotations
import contextlib import contextlib
import os
import socket import socket
import ssl import ssl
from ._protocol import ProtocolError
from .exceptions import InterfaceError, OperationalError from .exceptions import InterfaceError, OperationalError
def _max_read_bytes() -> int:
"""Ceiling for a single length-prefixed read, from IFX_MAX_READ_BYTES.
256 MiB by default: comfortably above any row, string table, or blob
chunk a real query produces, and far below the values a desynced
stream invents. A genuinely larger single value is possible (a very
large TEXT column read in one go), which is why the knob exists.
"""
raw = os.environ.get("IFX_MAX_READ_BYTES")
if raw:
try:
value = int(raw)
except ValueError:
value = 0
if value > 0:
return value
return 256 * 1024 * 1024
MAX_READ_BYTES = _max_read_bytes()
# A ``tls`` parameter to ``IfxSocket`` accepts: # A ``tls`` parameter to ``IfxSocket`` accepts:
# False (default) — plain TCP # False (default) — plain TCP
# True — TLS with verification disabled (dev / self-signed) # True — TLS with verification disabled (dev / self-signed)
@ -42,7 +66,16 @@ def _make_default_dev_context() -> ssl.SSLContext:
class IfxSocket: class IfxSocket:
"""Owns a connected TCP socket and provides exact-N read/write.""" """Owns a connected TCP socket and provides exact-N read/write.
Phase 39 also owns the read-ahead buffer for ``BufferedSocketReader``.
The buffer (``_recv_buf`` + ``_recv_pos``) is connection-scoped, not
reader-scoped readers are short-lived per-PDU views, but pipelined
responses (e.g., Phase 33's pipelined ``executemany``) can stream
multiple responses across reader boundaries. Persisting the buffer
here means we never throw away pre-fetched bytes when one reader
is destroyed and the next one is created.
"""
def __init__( def __init__(
self, self,
@ -58,6 +91,11 @@ class IfxSocket:
self._port = port self._port = port
self._read_timeout = read_timeout self._read_timeout = read_timeout
self._sock: socket.socket | None = None self._sock: socket.socket | None = None
# Phase 39 read-ahead buffer. Empty until a BufferedSocketReader
# touches us; subsequent readers share this buffer state.
self._recv_buf: bytearray = bytearray()
self._recv_pos: int = 0
self._recv_size: int = 65536
try: try:
sock = socket.create_connection((host, port), timeout=connect_timeout) sock = socket.create_connection((host, port), timeout=connect_timeout)
@ -110,10 +148,42 @@ class IfxSocket:
raise OperationalError(f"write failed: {e}") from e raise OperationalError(f"write failed: {e}") from e
def read_exact(self, n: int) -> bytes: def read_exact(self, n: int) -> bytes:
"""Read exactly ``n`` bytes or raise on EOF / timeout.""" """Read exactly ``n`` bytes or raise on EOF / timeout.
Consumes the read-ahead buffer before touching the socket. This
matters because two readers share one stream: ``BufferedSocketReader``
fills ``_recv_buf`` (over-reading by design, up to ``_recv_size``),
while ``Connection._drain_to_eot``, ``_raise_sq_err`` and the
login path call this method directly. Recv'ing here while bytes
sat unconsumed in the buffer would skip them, and skipped bytes
in a length-framed protocol don't announce themselves — the next
read lands mid-field and every subsequent one is wrong.
No workload triggers it today: the server sends one response per
request, so recv returns exactly that response and the buffered
reader consumes all of it before control returns here. That is a
property of the traffic, not of the code. Pipelined executemany
already puts multiple responses in flight, and the buffer is
connection-scoped precisely so read-ahead can cross response
boundaries. Making the two paths agree by construction costs one
branch on a cold path.
"""
if self._sock is None: if self._sock is None:
raise InterfaceError("socket is closed") raise InterfaceError("socket is closed")
if n <= 0:
return b""
wanted = n
chunks: list[bytes] = [] chunks: list[bytes] = []
buffered = len(self._recv_buf) - self._recv_pos
if buffered > 0:
take = min(buffered, n)
chunks.append(
bytes(self._recv_buf[self._recv_pos : self._recv_pos + take])
)
self._recv_pos += take
n -= take
if n == 0:
return chunks[0]
remaining = n remaining = n
while remaining > 0: while remaining > 0:
try: try:
@ -124,12 +194,63 @@ class IfxSocket:
if not chunk: if not chunk:
self._force_close() self._force_close()
raise OperationalError( raise OperationalError(
f"server closed connection mid-read (wanted {n} bytes, got {n - remaining})" f"server closed connection mid-read "
f"(wanted {wanted} bytes, got {wanted - remaining})"
) )
chunks.append(chunk) chunks.append(chunk)
remaining -= len(chunk) remaining -= len(chunk)
return b"".join(chunks) return b"".join(chunks)
def fill_recv_buf(self, need: int) -> None:
"""Phase 39: ensure the read-ahead buffer holds ``need`` bytes
forward of ``_recv_pos``, recv'ing from the socket as needed.
Compaction: when the read cursor has advanced past
``_recv_size`` bytes, slice off the consumed prefix in place.
That bounds memory at roughly ``2 * recv_size`` while still
amortizing recv calls over many fields.
"""
if self._sock is None:
raise InterfaceError("socket is closed")
if need > MAX_READ_BYTES:
# ``need`` is almost always a length field straight off the
# wire. Trusting it means a corrupt or desynced stream turns
# into an allocation of whatever that field happened to say —
# a garbage 0x7FFFFFFF reads as a 2 GB request, and the loop
# below sits in recv until the read timeout while the buffer
# grows. Refusing names the number instead, which is the
# diagnostic: a length that absurd means framing is already
# lost upstream, not that the row is genuinely that big.
raise ProtocolError(
f"refusing to read {need} bytes in one field (limit "
f"{MAX_READ_BYTES}). Either the wire has desynced, or a "
f"single value really is this large — raise the limit "
f"with the IFX_MAX_READ_BYTES environment variable."
)
avail = len(self._recv_buf) - self._recv_pos
if avail >= need:
return
if self._recv_pos > self._recv_size:
del self._recv_buf[: self._recv_pos]
self._recv_pos = 0
avail = len(self._recv_buf)
target = max(need, self._recv_size)
sock = self._sock
while avail < need:
try:
chunk = sock.recv(target - avail)
except OSError as e:
self._force_close()
raise OperationalError(f"read failed: {e}") from e
if not chunk:
self._force_close()
raise OperationalError(
f"server closed connection mid-read "
f"(wanted {need} bytes, got {avail})"
)
self._recv_buf.extend(chunk)
avail += len(chunk)
def close(self) -> None: def close(self) -> None:
"""Close the socket. Idempotent and never raises.""" """Close the socket. Idempotent and never raises."""
if self._sock is None: if self._sock is None:

View File

@ -62,7 +62,10 @@ from __future__ import annotations
import asyncio import asyncio
import contextlib import contextlib
import functools import functools
import threading
import weakref
from collections.abc import AsyncIterator, Awaitable, Callable from collections.abc import AsyncIterator, Awaitable, Callable
from concurrent.futures import ThreadPoolExecutor
from typing import Any, TypeVar from typing import Any, TypeVar
from . import connect as _sync_connect from . import connect as _sync_connect
@ -79,6 +82,61 @@ def _to_thread(fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
return asyncio.to_thread(fn, *args, **kwargs) return asyncio.to_thread(fn, *args, **kwargs)
def _run_on(
executor: ThreadPoolExecutor,
fn: Callable[..., T],
*args: Any,
**kwargs: Any,
) -> Awaitable[T]:
"""Await ``fn`` on a specific executor rather than the loop's default."""
loop = asyncio.get_running_loop()
return loop.run_in_executor(executor, functools.partial(fn, *args, **kwargs))
def _connection_executor(conn: _SyncConnection) -> ThreadPoolExecutor:
"""The dedicated worker thread for one connection, created on demand.
``asyncio.to_thread`` runs on the event loop's default executor, which
the whole process shares and which is sized from the CPU count
``min(32, cpu_count + 4)``, so six threads on a two-CPU container.
Two consequences, both silent.
A cancelled await does not stop its worker. ``asyncio.to_thread``
cannot interrupt the thread, so it keeps running the wire call until
the read timeout expires. Cancellation is ordinary in a web app a
client disconnect cancels the request task so a handful of them
pins every thread in the shared pool and unrelated ``to_thread`` work
anywhere else in the process stops dead. Measured: six cancelled
calls against a six-worker default executor starve an unrelated
``to_thread`` indefinitely.
And pool concurrency was capped by the same number without saying so.
A pool with ``max_size=20`` on a two-CPU box ran six queries at once.
One thread per connection is the right size rather than a compromise:
the sync connection serializes every wire operation on its own lock,
so a second thread could do nothing but wait for the first. It also
rules out a deadlock that a shared pool-sized executor invites with
N threads and N connections, N tasks blocked in ``acquire`` occupy
every thread while the connection they are waiting for is held by a
task that now needs a thread of its own to finish and release it.
The executor lives on the sync connection so it survives being
returned to the pool and handed out again. The finalizer is the
backstop for a connection dropped without ``close()``: a
``ThreadPoolExecutor`` that is never shut down leaves its worker
parked on the work queue for the life of the process.
"""
executor = getattr(conn, "_async_executor", None)
if executor is None:
executor = ThreadPoolExecutor(
max_workers=1, thread_name_prefix="informix-conn"
)
conn._async_executor = executor
weakref.finalize(conn, executor.shutdown, wait=False)
return executor
class AsyncCursor: class AsyncCursor:
"""Async wrapper over a sync :class:`Cursor`. Each I/O call awaits """Async wrapper over a sync :class:`Cursor`. Each I/O call awaits
a thread-offloaded version of the sync operation. a thread-offloaded version of the sync operation.
@ -88,10 +146,14 @@ class AsyncCursor:
paying the thread-hop cost. paying the thread-hop cost.
""" """
__slots__ = ("_cur",) __slots__ = ("_cur", "_run")
def __init__(self, cur: _SyncCursor): def __init__(self, cur: _SyncCursor, run: Callable[..., Awaitable[Any]]):
self._cur = cur self._cur = cur
# The owning connection's runner, so cursor I/O lands on that
# connection's dedicated thread rather than the shared default
# executor. See _connection_executor.
self._run = run
# -- Pass-through synchronous attributes (no I/O) --------------------- # -- Pass-through synchronous attributes (no I/O) ---------------------
@ -120,32 +182,32 @@ class AsyncCursor:
async def execute( async def execute(
self, operation: str, parameters: Any = None self, operation: str, parameters: Any = None
) -> None: ) -> None:
await _to_thread(self._cur.execute, operation, parameters) await self._run(self._cur.execute, operation, parameters)
async def executemany( async def executemany(
self, operation: str, seq_of_parameters: Any self, operation: str, seq_of_parameters: Any
) -> None: ) -> None:
await _to_thread( await self._run(
self._cur.executemany, operation, list(seq_of_parameters) self._cur.executemany, operation, list(seq_of_parameters)
) )
async def fetchone(self) -> tuple | None: async def fetchone(self) -> tuple | None:
return await _to_thread(self._cur.fetchone) return await self._run(self._cur.fetchone)
async def fetchmany(self, size: int | None = None) -> list[tuple]: async def fetchmany(self, size: int | None = None) -> list[tuple]:
return await _to_thread(self._cur.fetchmany, size) return await self._run(self._cur.fetchmany, size)
async def fetchall(self) -> list[tuple]: async def fetchall(self) -> list[tuple]:
return await _to_thread(self._cur.fetchall) return await self._run(self._cur.fetchall)
async def close(self) -> None: async def close(self) -> None:
await _to_thread(self._cur.close) await self._run(self._cur.close)
# Phase 10/11 BLOB helpers (preserve the sync API surface) # Phase 10/11 BLOB helpers (preserve the sync API surface)
async def read_blob_column( async def read_blob_column(
self, sql: str, params: tuple = () self, sql: str, params: tuple = ()
) -> bytes | None: ) -> bytes | None:
return await _to_thread(self._cur.read_blob_column, sql, params) return await self._run(self._cur.read_blob_column, sql, params)
async def write_blob_column( async def write_blob_column(
self, self,
@ -155,7 +217,7 @@ class AsyncCursor:
*, *,
clob: bool = False, clob: bool = False,
) -> None: ) -> None:
await _to_thread( await self._run(
functools.partial( functools.partial(
self._cur.write_blob_column, self._cur.write_blob_column,
sql, blob_data, params, clob=clob, sql, blob_data, params, clob=clob,
@ -177,14 +239,23 @@ class AsyncCursor:
class AsyncConnection: class AsyncConnection:
"""Async wrapper over a sync :class:`Connection`.""" """Async wrapper over a sync :class:`Connection`."""
__slots__ = ("_conn",) __slots__ = ("_conn", "_executor")
def __init__(self, conn: _SyncConnection): def __init__(self, conn: _SyncConnection):
self._conn = conn self._conn = conn
self._executor = _connection_executor(conn)
def _run(self, fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
"""Run a blocking connection call on this connection's own thread."""
return _run_on(self._executor, fn, *args, **kwargs)
@classmethod @classmethod
async def connect(cls, *args: Any, **kwargs: Any) -> AsyncConnection: async def connect(cls, *args: Any, **kwargs: Any) -> AsyncConnection:
"""Open a connection. Same parameters as :func:`informix_db.connect`.""" """Open a connection. Same parameters as :func:`informix_db.connect`.
The connect itself still goes to the default executor -- there is
no connection yet to own a thread, and it is one bounded call.
"""
sync_conn = await _to_thread( sync_conn = await _to_thread(
functools.partial(_sync_connect, *args, **kwargs) functools.partial(_sync_connect, *args, **kwargs)
) )
@ -195,22 +266,46 @@ class AsyncConnection:
return self._conn.closed return self._conn.closed
async def cursor(self) -> AsyncCursor: async def cursor(self) -> AsyncCursor:
sync_cur = await _to_thread(self._conn.cursor) sync_cur = await self._run(self._conn.cursor)
return AsyncCursor(sync_cur) return AsyncCursor(sync_cur, self._run)
async def commit(self) -> None: async def commit(self) -> None:
await _to_thread(self._conn.commit) await self._run(self._conn.commit)
async def rollback(self) -> None: async def rollback(self) -> None:
await _to_thread(self._conn.rollback) await self._run(self._conn.rollback)
async def close(self) -> None: async def close(self) -> None:
await _to_thread(self._conn.close) """Close the connection and stop the thread that served it.
One thread per connection is only affordable if the thread goes
away with the connection. ``ThreadPoolExecutor`` workers park on
the work queue rather than exiting when idle, so an executor that
is never shut down leaks its thread for the life of the process
the ``weakref.finalize`` in ``_connection_executor`` is a backstop
for connections dropped without ``close()``, not a substitute for
closing here.
``wait=False`` because we are on the event loop: the worker has
just finished the close and needs no waiting, and blocking the
loop to confirm that would be the one thing this module exists to
avoid.
"""
try:
await self._run(self._conn.close)
finally:
self._executor.shutdown(wait=False)
# Drop the reference too. A shut-down executor rejects new
# work, so leaving it attached would turn any later wrap of
# this sync connection into a RuntimeError rather than a
# fresh thread.
with contextlib.suppress(AttributeError):
del self._conn._async_executor
async def fast_path_call( async def fast_path_call(
self, signature: str, *params: object self, signature: str, *params: object
) -> list[object]: ) -> list[object]:
return await _to_thread(self._conn.fast_path_call, signature, *params) return await self._run(self._conn.fast_path_call, signature, *params)
# Async context-manager support # Async context-manager support
async def __aenter__(self) -> AsyncConnection: async def __aenter__(self) -> AsyncConnection:
@ -249,14 +344,69 @@ class AsyncConnectionPool:
return self._pool.idle_count return self._pool.idle_count
async def acquire(self, timeout: float | None = None) -> AsyncConnection: async def acquire(self, timeout: float | None = None) -> AsyncConnection:
sync_conn = await _to_thread(self._pool.acquire, timeout) """Acquire a connection, surviving cancellation of the waiter.
``asyncio.to_thread`` cannot interrupt its worker. If the task
awaiting an acquire is cancelled while the worker is still
blocked waiting for a free connection, the worker eventually
succeeds and hands back a connection **that nobody owns** it
is checked out of the pool and never returned. Repeat that and
the pool starves to death.
This is not hypothetical for anyone serving HTTP: a client
disconnecting cancels the request task, and under load those
cancellations land precisely while waiting for a connection. The
pool then dies one slot at a time, and only under load.
So the inner future is shielded cancelling the caller must not
orphan a result we still need to see and if the caller does go
away, a callback returns whatever the worker produced to the
pool. ``add_done_callback`` fires immediately when the future has
already resolved, so the "worker finished a moment before the
cancellation" race is covered by the same code path.
"""
fut = asyncio.ensure_future(_to_thread(self._pool.acquire, timeout))
try:
sync_conn = await asyncio.shield(fut)
except asyncio.CancelledError:
fut.add_done_callback(self._return_orphan)
raise
return AsyncConnection(sync_conn) return AsyncConnection(sync_conn)
def _return_orphan(self, fut: asyncio.Future) -> None:
"""Give back a connection whose acquirer was cancelled.
Runs on the event loop, so it must not block: ``release`` takes
the connection's wire lock and can wait. Hand off to a short-lived
daemon thread rather than the running loop this path also has to
work while the loop is shutting down, which is exactly when
``create_task`` is unavailable.
"""
if fut.cancelled() or fut.exception() is not None:
return
conn = fut.result()
def _release() -> None:
# broken=False: the connection was never handed to anyone, so
# its wire is untouched and it is safe to reuse. Evicting here
# would trade a leak for needless reconnect churn.
with contextlib.suppress(Exception):
self._pool.release(conn, broken=False)
threading.Thread(target=_release, daemon=True).start()
async def release( async def release(
self, conn: AsyncConnection, *, broken: bool = False self, conn: AsyncConnection, *, broken: bool = False
) -> None: ) -> None:
await _to_thread( # On the connection's own thread, not the default executor. That
functools.partial(self._pool.release, conn._conn, broken=broken) # thread is idle by definition — the caller is done with the
# connection — and it keeps the release off a shared pool that
# tasks blocked in ``acquire`` may have filled. Release has to
# win that race: it is what frees the connection they are
# waiting for.
await _run_on(
conn._executor,
functools.partial(self._pool.release, conn._conn, broken=broken),
) )
@contextlib.asynccontextmanager @contextlib.asynccontextmanager

View File

@ -12,15 +12,18 @@ reference in ``docs/CAPTURES/01-connect-only.socat.log``.
from __future__ import annotations from __future__ import annotations
import contextlib import contextlib
import logging
import os import os
import socket as socket_mod import socket as socket_mod
import ssl import ssl
import struct import struct
import threading import threading
import weakref
from io import BytesIO from io import BytesIO
from pathlib import Path from pathlib import Path
from . import _auth from . import _auth
from ._capabilities import CLIENT_PROTOCOLS_MASK, ServerCapabilities
from ._messages import ( from ._messages import (
APPL_ID, APPL_ID,
APPL_TYPE, APPL_TYPE,
@ -36,10 +39,16 @@ from ._messages import (
SLHeader, SLHeader,
StmtOptions, StmtOptions,
) )
from ._protocol import IfxStreamReader, IfxStreamWriter, ProtocolError, make_pdu_writer from ._protocol import (
WIRE_ERRORS,
IfxStreamReader,
IfxStreamWriter,
ProtocolError,
make_pdu_writer,
)
from ._socket import IfxSocket from ._socket import IfxSocket
from .cursors import Cursor from .cursors import Cursor
from .exceptions import InterfaceError, OperationalError from .exceptions import InterfaceError, OperationalError, ProgrammingError
# Default capability bits the JDBC reference sends. Validated against # Default capability bits the JDBC reference sends. Validated against
# 01-connect-only.socat.log via the PDU diff in tests/test_pdu_match.py: # 01-connect-only.socat.log via the PDU diff in tests/test_pdu_match.py:
@ -69,6 +78,9 @@ _LOCALE_ENCODING_MAP = {
} }
_log = logging.getLogger(__name__)
def _extract_server_error_text(payload: bytes) -> str | None: def _extract_server_error_text(payload: bytes) -> str | None:
"""Pull the longest printable run out of an opaque rejection payload. """Pull the longest printable run out of an opaque rejection payload.
@ -113,6 +125,74 @@ def _python_encoding_from_locale(locale: str) -> str:
return _LOCALE_ENCODING_MAP.get(suffix, "iso-8859-1") return _LOCALE_ENCODING_MAP.get(suffix, "iso-8859-1")
def _decode_conacc(rest: bytes) -> dict | None:
"""Decode the SLTYPE_CONACC server-metadata block.
Mirrors ``com.informix.asf.Connection.DecodeAscBinary``. ``rest`` is
the login response with its 2-byte total-length prefix already
stripped, i.e. starting at the SLType byte.
Layout, verified byte-for-byte against Informix 12.10, 14.10 and 15::
byte SLType (2 = CONACC)
3 skip
short 100 marker
short 101 marker
4 skip
short len; skip len (codeset id, e.g. "IEEEI")
short 108 marker JDBC hard-fails if this isn't 108
12 skip
short len; bytes server version string
short len; bytes serial number
short len; bytes applid name
int Cap_1 client protocol level, echoed (316 = enhanced)
int Cap_2
int Cap_3
Returns ``None`` if the block doesn't match that shape. Callers must
treat that as "no capability info", never as a connection failure
every server we support connects fine without any of this being
decodable, and a future server variant should degrade, not break.
"""
try:
offset = 1 + 3 # SLType + 3 skipped bytes
marker_a, marker_b = struct.unpack_from("!hh", rest, offset)
offset += 4
if marker_a != 100 or marker_b != 101:
return None
offset += 4
(codeset_len,) = struct.unpack_from("!h", rest, offset)
offset += 2 + codeset_len
(marker_c,) = struct.unpack_from("!h", rest, offset)
offset += 2
if marker_c != 108:
return None
offset += 12
strings: list[str] = []
for _ in range(3):
(length,) = struct.unpack_from("!h", rest, offset)
offset += 2
if length < 0:
return None
strings.append(
rest[offset : offset + length].rstrip(b"\x00").decode(
"iso-8859-1", "replace"
)
)
offset += length
cap_1, cap_2, cap_3 = struct.unpack_from("!iii", rest, offset)
except (struct.error, IndexError, UnicodeDecodeError):
return None
return {
"server_version": strings[0],
"serial_number": strings[1],
"applid_name": strings[2],
"cap_1": cap_1,
"cap_2": cap_2,
"cap_3": cap_3,
}
# Default environment variables sent in the login PDU (SQ_ASCENV section). # Default environment variables sent in the login PDU (SQ_ASCENV section).
# These match what the JDBC driver sends for a vanilla en_US.8859-1 # These match what the JDBC driver sends for a vanilla en_US.8859-1
# connection. Anything missing makes the server fall back to defaults. # connection. Anything missing makes the server fall back to defaults.
@ -126,6 +206,62 @@ _DEFAULT_ENV: dict[str, str] = {
} }
class _WireLock:
"""A reentrant lock that can say whether *this* thread already holds it.
``threading.RLock`` cannot answer that question in public API, and
the difference is not academic. The cursor finalizer runs at GC time
on whatever thread happened to allocate including a thread that is
at that moment mid-statement holding this lock. It probes with
``acquire(blocking=False)`` intending to mean "is anyone using the
wire?", but an RLock grants a reentrant acquire to its own owner, so
the probe returns True and the finalizer sends CLOSE/RELEASE into the
middle of the statement it interrupted. The victim gets ``-208``.
Reachability is not exotic. Refcounting frees a dropped cursor
immediately, before the next statement, which is why this went
unnoticed but a cursor caught in a reference cycle waits for a
collection instead, and cycles are ordinary in Python. Any traceback
that holds a cursor makes one.
``held_by_current_thread`` reads ``_depth``/``_owner`` without the
lock, which is safe: both are mutated only under it, and the only
values another thread can leave behind are a depth of zero or an
owner that isn't us. Either way the answer is False, correctly.
"""
__slots__ = ("_depth", "_lock", "_owner")
def __init__(self) -> None:
self._lock = threading.RLock()
self._owner: int | None = None
self._depth = 0
def acquire(self, blocking: bool = True, timeout: float = -1) -> bool:
acquired = self._lock.acquire(blocking, timeout)
if acquired:
self._owner = threading.get_ident()
self._depth += 1
return acquired
def release(self) -> None:
self._depth -= 1
if self._depth == 0:
self._owner = None
self._lock.release()
def __enter__(self) -> _WireLock:
self.acquire()
return self
def __exit__(self, *exc_info: object) -> None:
self.release()
@property
def held_by_current_thread(self) -> bool:
return self._depth > 0 and self._owner == threading.get_ident()
class Connection: class Connection:
"""A SQLI session. Owns one TCP socket and the post-login state. """A SQLI session. Owns one TCP socket and the post-login state.
@ -147,6 +283,7 @@ class Connection:
client_locale: str = "en_US.8859-1", client_locale: str = "en_US.8859-1",
env: dict[str, str] | None = None, env: dict[str, str] | None = None,
autocommit: bool = False, # honored from Phase 3 onward autocommit: bool = False, # honored from Phase 3 onward
row_factory: object | None = None,
tls: bool | ssl.SSLContext = False, tls: bool | ssl.SSLContext = False,
tls_server_hostname: str | None = None, tls_server_hostname: str | None = None,
): ):
@ -170,7 +307,7 @@ class Connection:
# this lock with a timeout, then calls ``conn.rollback()`` — # this lock with a timeout, then calls ``conn.rollback()`` —
# which itself acquires the lock. Same thread, two acquires. # which itself acquires the lock. Same thread, two acquires.
# Reentrance must be cheap and correct. # Reentrance must be cheap and correct.
self._wire_lock = threading.RLock() self._wire_lock = _WireLock()
# Phase 29: deferred-cleanup queue for cursor finalizers that # Phase 29: deferred-cleanup queue for cursor finalizers that
# couldn't acquire the wire lock at GC time. Each entry is a # couldn't acquire the wire lock at GC time. Each entry is a
# PDU's worth of bytes (typically a CLOSE or RELEASE) that # PDU's worth of bytes (typically a CLOSE or RELEASE) that
@ -190,12 +327,21 @@ class Connection:
# under ``_wire_lock``. # under ``_wire_lock``.
self._pending_cleanup: list[bytes] = [] self._pending_cleanup: list[bytes] = []
self._cleanup_lock = threading.Lock() self._cleanup_lock = threading.Lock()
# Weak ref to the scrollable cursor currently holding the
# session's statement slot, or None. Weak so that abandoning a
# scrollable cursor still lets its finalizer run — a strong ref
# here would keep the very object alive whose GC we depend on.
self._open_scroll_cursor: weakref.ref | None = None
# Logged-DB transaction state: True iff there's an open server-side # Logged-DB transaction state: True iff there's an open server-side
# transaction (SQ_BEGIN sent, not yet committed/rolled-back). The # transaction (SQ_BEGIN sent, not yet committed/rolled-back). The
# cursor uses this to decide whether to send an implicit SQ_BEGIN # cursor uses this to decide whether to send an implicit SQ_BEGIN
# before the next DML in non-autocommit mode. We default to "no # before the next DML in non-autocommit mode. We default to "no
# open txn" — the first DML will trigger SQ_BEGIN. # open txn" — the first DML will trigger SQ_BEGIN.
self._in_transaction = False self._in_transaction = False
# Default row type for cursors from this connection. None means
# plain tuples, which is the zero-cost default; see
# informix_db.rows for the opt-in named-access type.
self.row_factory = row_factory
# Tri-state: True after first successful SQ_BEGIN, False after # Tri-state: True after first successful SQ_BEGIN, False after
# an unlogged-DB rejection (-201). None until we've tried. # an unlogged-DB rejection (-201). None until we've tried.
# Used to avoid repeatedly probing on unlogged DBs. # Used to avoid repeatedly probing on unlogged DBs.
@ -205,6 +351,16 @@ class Connection:
# SQ_GETROUTINE; subsequent calls skip that round-trip. # SQ_GETROUTINE; subsequent calls skip that round-trip.
self._fp_handle_cache: dict[str, tuple[str, int]] = {} self._fp_handle_cache: dict[str, tuple[str, int]] = {}
# Server metadata from the login response and the SQ_PROTOCOLS
# negotiation. Both are populated during connect; both degrade to
# None rather than failing the connection.
self._conacc: dict | None = None
self._server_protocols: bytes | None = None
self._capabilities: ServerCapabilities | None = None
# Lazily filled by the server_version property. None = not asked
# yet; "" = asked and failed (don't retry on every access).
self._server_version_full: str | None = None
# Build the env-var dict sent in the login PDU. # Build the env-var dict sent in the login PDU.
self._env = dict(_DEFAULT_ENV) self._env = dict(_DEFAULT_ENV)
self._env["CLIENT_LOCALE"] = client_locale self._env["CLIENT_LOCALE"] = client_locale
@ -274,22 +430,80 @@ class Connection:
raise InterfaceError("connection is closed") raise InterfaceError("connection is closed")
return Cursor(self, scrollable=scrollable) return Cursor(self, scrollable=scrollable)
def _send_pdu(self, pdu: bytes) -> None: def _send_pdu(self, pdu: bytes, *, statement_boundary: bool = False) -> None:
"""Send an assembled PDU. Used by Cursor. """Send an assembled PDU. Used by Cursor.
Phase 29: opportunistically drains any pending cleanup PDUs Caller must hold ``_wire_lock`` (every actual call site does).
from the deferred-cleanup queue *before* sending the new PDU.
Caller must hold ``_wire_lock`` (every actual call site already ``statement_boundary=True`` additionally drains the deferred
does execute/executemany/_sfetch_at, commit, rollback, cleanup queue first. **Only pass it when this PDU is the first
fast_path_call, etc.). The drain happens under that lock so of a new statement**, meaning no statement is currently open
the queued cleanup atomically completes before the next op. server-side.
The queue previously drained before *every* PDU, which was wrong
in a way that took a repro to see. ``SQ_CLOSE`` and
``SQ_RELEASE`` carry no statement identifier they act on the
server's *current* statement. A finalizer enqueues precisely
because it lost the race for the wire lock, which means another
thread is mid-statement; that thread's next ``_send_pdu`` was
then its own ``CURNAME``/``NFETCH``, and the drain released the
statement out from under it. The caller saw a nonsense error on
valid SQL: ``-208`` when injected before the first fetch,
``-267`` "transaction has been rolled back" between fetch
batches. Both point nowhere near the actual cause, and the
window is exactly the window in which enqueueing happens.
Draining only at a boundary costs nothing: cleanup that misses
one statement is picked up by the next.
""" """
if self._closed: if self._closed:
raise InterfaceError("connection is closed") raise InterfaceError("connection is closed")
if self._pending_cleanup: if statement_boundary and self._pending_cleanup:
self._drain_pending_cleanup() self._drain_pending_cleanup()
self._sock.write_all(pdu) self._sock.write_all(pdu)
def _check_scroll_cursor_conflict(self, requester: object) -> None:
"""Refuse to start a statement while a scrollable cursor is open.
A server-side scrollable cursor occupies the session's statement
slot, and SQLI gives us no way to address around it: ``SQ_CLOSE``,
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on the session's current
statement.
The server does not refuse politely. Starting another statement
returns ``-285``, and the scrollable cursor is collateral damage
its next fetch comes back ``-267`` "the transaction has been
rolled back, all locks released". Two unattributable failures
from code that looks entirely ordinary: iterate a large result
set with a scrollable cursor, run a lookup query partway through.
Multiplexing is presumably expressible JDBC prefixes every
statement-scoped PDU with the statement id, which the server does
assign distinctly (0 and 1 for two concurrent cursors). But the
id alone is not sufficient: addressing the second cursor's
``SQ_SFETCH`` to its own id returns ``-259`` "cursor not open".
Until that is understood, refusing is the honest behaviour. It
costs the caller a second connection and it never corrupts.
"""
ref = self._open_scroll_cursor
if ref is None:
return
other = ref()
if (
other is None
or other is requester
or not getattr(other, "_server_cursor_open", False)
):
self._open_scroll_cursor = None
return
raise ProgrammingError(
"a scrollable cursor is open on this connection; Informix "
"allows only one statement per session, so running another "
"statement here would fail with -285 and destroy the "
"scrollable cursor as well. Close the scrollable cursor "
"first, or use a separate connection for the other statement."
)
def _enqueue_cleanup(self, pdus: list[bytes]) -> None: def _enqueue_cleanup(self, pdus: list[bytes]) -> None:
"""Append cleanup PDUs to the deferred queue. """Append cleanup PDUs to the deferred queue.
@ -319,7 +533,6 @@ class Connection:
discarded; the server-side resources they would have released discarded; the server-side resources they would have released
are freed when the session ends anyway. are freed when the session ends anyway.
""" """
from ._protocol import ProtocolError
with self._cleanup_lock: with self._cleanup_lock:
if not self._pending_cleanup: if not self._pending_cleanup:
@ -330,11 +543,42 @@ class Connection:
try: try:
self._sock.write_all(pdu) self._sock.write_all(pdu)
self._drain_to_eot() self._drain_to_eot()
except (ProtocolError, OSError, OperationalError): except Exception as exc:
# Wire is unrecoverable; force-close. Subsequent if getattr(exc, "sqlcode", None) is not None:
# ``_send_pdu`` will raise InterfaceError. Server # The *server* rejected the cleanup — a stale entry
# cleanup of the remaining queued entries happens # for a cursor it no longer has. Queued cleanup goes
# implicitly at session end. # stale routinely: the finalizer enqueues, then the
# cursor gets closed properly before the drain runs.
#
# This is not a wire problem. ``_raise_sq_err``
# self-drains the trailing SQ_EOT, so the wire is
# still aligned and the remaining entries are still
# worth sending.
#
# It must not escape, and it must not be treated as
# fatal. Both were wrong before: a stale CLOSE draws
# ``-267``, which is an OperationalError, which is in
# WIRE_ERRORS — so a stale queue entry force-closed a
# perfectly healthy connection. And this runs at the
# start of somebody else's statement, so letting it
# out would fail their good SQL with an error about a
# cursor they never opened.
_log.debug(
"deferred cleanup rejected by server (stale entry), "
"continuing: %r",
exc,
)
continue
if not isinstance(exc, WIRE_ERRORS):
_log.warning(
"unexpected error draining deferred cleanup: %r", exc
)
# No sqlcode means the failure came from the wire, not
# the server: the socket died, or framing desynced and we
# can no longer say where a response ends. Force-close.
# Subsequent ``_send_pdu`` raises InterfaceError. The
# server-side resources the remaining entries would have
# freed are released when the session ends anyway.
self._closed = True self._closed = True
with contextlib.suppress(Exception): with contextlib.suppress(Exception):
self._sock.close() self._sock.close()
@ -513,7 +757,7 @@ class Connection:
# method but stable across versions; cheap (~50ns) and only # method but stable across versions; cheap (~50ns) and only
# checks the current thread. If it ever changes shape, drop # checks the current thread. If it ever changes shape, drop
# this assert — the doc still names the precondition. # this assert — the doc still names the precondition.
assert self._wire_lock._is_owned(), ( assert self._wire_lock.held_by_current_thread, (
"_ensure_transaction called without _wire_lock held; " "_ensure_transaction called without _wire_lock held; "
"the cursor method that called it must wrap its body in " "the cursor method that called it must wrap its body in "
"`with self._conn._wire_lock:`" "`with self._conn._wire_lock:`"
@ -589,9 +833,9 @@ class Connection:
# The 8-byte protocols mask is the JDBC reference value from # The 8-byte protocols mask is the JDBC reference value from
# docs/CAPTURES/02-select-1.socat.log; we replay it verbatim # docs/CAPTURES/02-select-1.socat.log; we replay it verbatim
# since the bits are opaque (server-recognized features). # since the bits are opaque (server-recognized features).
protocols_mask = bytes.fromhex("fffc7ffc3c8caa97") self._send_protocols(CLIENT_PROTOCOLS_MASK)
self._send_protocols(protocols_mask) self._drain_to_eot() # captures the reply into self._server_protocols
self._drain_to_eot() self._build_capabilities()
# Step 2: SQ_INFO with INFO_ENV subtype + session env vars. # Step 2: SQ_INFO with INFO_ENV subtype + session env vars.
# The actual on-wire format (from JDBC's sendEnv at IfxSqli.java # The actual on-wire format (from JDBC's sendEnv at IfxSqli.java
@ -626,6 +870,54 @@ class Connection:
self._send_dbopen(self._database) self._send_dbopen(self._database)
self._drain_to_eot() self._drain_to_eot()
def _build_capabilities(self) -> None:
"""Assemble ``ServerCapabilities`` from the login response and the
SQ_PROTOCOLS reply, and warn about contradicted assumptions.
This is observation only nothing in the parse paths branches on
these bits yet. The value is diagnostic: this driver hardcodes
several wire-framing choices that SQLI actually negotiates, and
those choices are correct on every server we have measured
(Informix 12.10, 14.10, 15). On some server we have not measured
they might not be, and the failure mode for a framing mismatch is
silently corrupted rows, which is about the worst way for a
database driver to fail. A log warning naming the specific bit
turns that into something diagnosable from a bug report.
Deliberately never raises: a server that declines to negotiate is
not necessarily broken, and refusing the connection over a
diagnostic would be a worse outcome than proceeding.
"""
if self._server_protocols is None:
return
conacc = self._conacc or {}
try:
caps = ServerCapabilities.from_wire(
self._server_protocols,
cap_1=conacc.get("cap_1", 0),
cap_2=conacc.get("cap_2", 0),
cap_3=conacc.get("cap_3", 0),
server_version=conacc.get("server_version", ""),
serial_number=conacc.get("serial_number", ""),
applid_name=conacc.get("applid_name", ""),
)
except Exception:
_log.debug("could not decode server capabilities", exc_info=True)
return
self._capabilities = caps
problems = caps.violated_assumptions()
if problems:
_log.warning(
"Informix server %r negotiated capabilities that contradict "
"this driver's hardcoded wire framing (mask=%s). Result rows "
"may be decoded incorrectly. Please report this with the "
"mask and your server version. Details: %s",
caps.server_version or "<unknown version>",
caps.raw_mask.hex(),
"; ".join(problems),
)
def _send_protocols(self, protocols: bytes) -> None: def _send_protocols(self, protocols: bytes) -> None:
"""Emit a SQ_PROTOCOLS PDU per ``IfxSqli.sendProtocols``. """Emit a SQ_PROTOCOLS PDU per ``IfxSqli.sendProtocols``.
@ -668,9 +960,13 @@ class Connection:
elif tag == MessageType.SQ_PROTOCOLS: elif tag == MessageType.SQ_PROTOCOLS:
# ``[short payloadLen][bytes payload][byte 0 if odd-len pad]`` # ``[short payloadLen][bytes payload][byte 0 if odd-len pad]``
# Then the loop continues and consumes the next tag (usually SQ_EOT). # Then the loop continues and consumes the next tag (usually SQ_EOT).
# The payload is the negotiated 64-bit feature bitmap; stash
# it for _capabilities decoding. Reading it was always
# necessary for stream alignment — we just used to throw it
# away instead of looking at it.
payload_len = struct.unpack("!h", self._sock.read_exact(2))[0] payload_len = struct.unpack("!h", self._sock.read_exact(2))[0]
if payload_len > 0: if payload_len > 0:
self._sock.read_exact(payload_len) self._server_protocols = self._sock.read_exact(payload_len)
if payload_len & 1: if payload_len & 1:
self._sock.read_exact(1) # writePadded's even-alignment pad self._sock.read_exact(1) # writePadded's even-alignment pad
elif tag == MessageType.SQ_DONE: elif tag == MessageType.SQ_DONE:
@ -700,7 +996,6 @@ class Connection:
[short near_token_len][bytes name][optional pad][short SQ_EOT] [short near_token_len][bytes name][optional pad][short SQ_EOT]
""" """
from . import _errcodes from . import _errcodes
from ._protocol import ProtocolError
sqlcode = struct.unpack("!h", self._sock.read_exact(2))[0] sqlcode = struct.unpack("!h", self._sock.read_exact(2))[0]
isamcode = struct.unpack("!h", self._sock.read_exact(2))[0] isamcode = struct.unpack("!h", self._sock.read_exact(2))[0]
@ -716,7 +1011,7 @@ class Connection:
if name_len & 1: if name_len & 1:
self._sock.read_exact(1) self._sock.read_exact(1)
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace") near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
except (ProtocolError, OSError): except WIRE_ERRORS:
pass pass
# Phase 28: drain failure means wire desync — force-close so # Phase 28: drain failure means wire desync — force-close so
# subsequent operations don't inherit the broken state. # subsequent operations don't inherit the broken state.
@ -730,7 +1025,7 @@ class Connection:
next_tag = struct.unpack("!h", self._sock.read_exact(2))[0] next_tag = struct.unpack("!h", self._sock.read_exact(2))[0]
if next_tag == MessageType.SQ_EOT: if next_tag == MessageType.SQ_EOT:
break break
except (ProtocolError, OSError, OperationalError): except WIRE_ERRORS:
self._closed = True self._closed = True
with contextlib.suppress(Exception): with contextlib.suppress(Exception):
self._sock.close() self._sock.close()
@ -883,6 +1178,72 @@ class Connection:
# -- response parsing ------------------------------------------------- # -- response parsing -------------------------------------------------
@property
def server_capabilities(self):
"""The negotiated ``SQ_PROTOCOLS`` feature bitmap, or ``None``.
``None`` means the negotiation reply couldn't be decoded — the
connection still works, we just have no capability information.
See ``informix_db._capabilities.ServerCapabilities``.
"""
return self._capabilities
@property
def server_version_internal(self) -> str:
"""The version string carried in the login response.
This is Informix's *internal* protocol version, not the release
you installed: 12.10 reports ``9.56``, 14.10 reports ``9.59``,
and 15 reports ``15.0.1.0.3``. At the protocol level the two
older releases really are 9.x servers, which is why all three
speak an identical SQLI dialect.
Free it arrives with the login response. For the release
number, use :attr:`server_version`.
"""
return self._conacc.get("server_version", "") if self._conacc else ""
@property
def server_version(self) -> str:
"""The server's release version, e.g.
``'IBM Informix Dynamic Server Version 12.10.FC6'``.
**Performs one query on first access**, then caches the result
for the life of the connection. The login response only carries
the internal protocol version (see
:attr:`server_version_internal`), which reads as a wrong answer
a 12.10 server announces itself as 9.56 so the release
number has to be asked for with ``DBINFO('version','full')``.
Falls back to the internal string if the query fails, which it
can when no database is open. Never raises: a version lookup
should not be able to break a working connection.
"""
if self._server_version_full is None:
self._server_version_full = self._query_server_version()
return self._server_version_full or self.server_version_internal
def _query_server_version(self) -> str:
"""One-shot ``DBINFO`` lookup for the release version.
Deliberately not done during connect: it costs a round-trip that
most callers never need, and connect latency is on the hot path
for pooled workloads.
"""
try:
cur = self.cursor()
try:
cur.execute(
"SELECT FIRST 1 DBINFO('version','full') FROM systables"
)
row = cur.fetchone()
finally:
cur.close()
except Exception:
_log.debug("could not query server version", exc_info=True)
return ""
return str(row[0]).strip() if row and row[0] else ""
def _parse_login_response(self) -> None: def _parse_login_response(self) -> None:
"""Read and parse the server's login response. """Read and parse the server's login response.
@ -916,8 +1277,12 @@ class Connection:
) )
elif sl_type != SLHeader.SLTYPE_CONACC: elif sl_type != SLHeader.SLTYPE_CONACC:
raise ProtocolError(f"unknown SLType in login response: {sl_type}") raise ProtocolError(f"unknown SLType in login response: {sl_type}")
# SLTYPE_CONACC — connection accepted. We don't (yet) decode the
# full server-side metadata. Phase 1 just needs to know "we got in". # SLTYPE_CONACC — connection accepted. Decode the server metadata
# block for the capability fields. Best-effort: a decode failure
# here must not fail an otherwise-good connection, since every
# server we've tested connects fine without any of this.
self._conacc = _decode_conacc(rest)
def _raise_from_rejection(self, reader: IfxStreamReader) -> None: def _raise_from_rejection(self, reader: IfxStreamReader) -> None:
"""Best-effort decode of the connection-rejection error block. """Best-effort decode of the connection-rejection error block.

View File

@ -204,8 +204,25 @@ def _decode_float(raw: bytes) -> float | None:
return _UNPACK_DOUBLE(raw)[0] return _UNPACK_DOUBLE(raw)[0]
def _decode_char(raw: bytes, encoding: str = "iso-8859-1") -> str: def _decode_char(raw: bytes, encoding: str = "iso-8859-1") -> str | None:
"""Strip trailing spaces (CHAR is space-padded to declared length).""" """Decode CHAR / NCHAR: fixed width, space-padded to the declared length.
A leading ``0x00`` is Informix's NULL marker for these types. Without
that check a NULL CHAR came back as ``''``, indistinguishable from a
genuinely empty one and ``WHERE c IS NULL`` disagreeing with what
the driver hands you is a nasty thing to debug. Wire evidence for
``CHAR(6)``::
'ab' -> 61 62 20 20 20 20
'' -> 20 20 20 20 20 20 all spaces
NULL -> 00 20 20 20 20 20 leading nul
The two are distinguishable, so we distinguish them. A real value
cannot begin with a nul: character data has no use for one, and the
server reserves it precisely as this marker.
"""
if raw[:1] == b"\x00":
return None
return raw.rstrip(b" \x00").decode(encoding) return raw.rstrip(b" \x00").decode(encoding)
@ -227,6 +244,41 @@ def _decode_bool(raw: bytes) -> bool:
return raw[0] in (ord("t"), ord("T"), 1) return raw[0] in (ord("t"), ord("T"), 1)
def _decode_int8(raw: bytes) -> int | None:
"""INT8 / SERIAL8 — the *legacy* 64-bit integer, 10 bytes on the wire.
NOT the same as BIGINT (52) / BIGSERIAL (53), which are plain 8-byte
big-endian. The INT8 layout mirrors ``ifx_int8_t`` and splits the
magnitude across two 32-bit halves in the opposite order you'd guess:
bytes 0-1 sign word: 0 = NULL, 1 = positive, -1 (0xFFFF) = negative
bytes 2-5 LOW 32 bits (unsigned, big-endian)
bytes 6-9 HIGH 32 bits (unsigned, big-endian)
Verified against Informix 12.10.FC12W1DE and 15.0.1.0.3DE, which emit
byte-identical encodings::
123456789012 -> 00 01 | be 99 1a 14 | 00 00 00 1c
-123456789012 -> ff ff | be 99 1a 14 | 00 00 00 1c
42 -> 00 01 | 00 00 00 2a | 00 00 00 00
NULL -> 00 00 | 00 00 00 00 | 00 00 00 00
Note the magnitude bytes are identical for +n and -n the sign lives
entirely in the leading word, so this is sign-magnitude, not two's
complement. Decoding it as a signed 64-bit integer gives the wrong
answer for every negative value.
"""
if len(raw) < 10:
raise ValueError(f"INT8 payload too short: {len(raw)} bytes, need 10")
sign = _UNPACK_SHORT(raw[0:2])[0]
if sign == 0:
return None
low = int.from_bytes(raw[2:6], "big", signed=False)
high = int.from_bytes(raw[6:10], "big", signed=False)
value = (high << 32) | low
return -value if sign < 0 else value
def _decode_date(raw: bytes) -> datetime.date | None: def _decode_date(raw: bytes) -> datetime.date | None:
"""4-byte big-endian signed int = day count from 1899-12-31. NULL = 0x80000000.""" """4-byte big-endian signed int = day count from 1899-12-31. NULL = 0x80000000."""
days = _UNPACK_INT(raw)[0] days = _UNPACK_INT(raw)[0]
@ -534,6 +586,13 @@ FIXED_WIDTHS: dict[int, int] = {
IfxType.BIGSERIAL: 8, IfxType.BIGSERIAL: 8,
IfxType.DATE: 4, IfxType.DATE: 4,
IfxType.BOOL: 1, IfxType.BOOL: 1,
# INT8/SERIAL8 are fixed-width at 10 bytes — NOT 8. See _decode_int8.
# Omitting these was a silent-corruption bug: they fell through to the
# unknown-type path, which surfaces ``encoded_length`` raw bytes.
# ``encoded_length`` happens to be 10 for INT8, so the stream stayed
# aligned and the only symptom was a bytes object where an int belonged.
IfxType.INT8: 10,
IfxType.SERIAL8: 10,
} }
@ -557,6 +616,8 @@ DECODERS: dict[int, DecoderFn] = {
IfxType.SERIAL: _decode_int, IfxType.SERIAL: _decode_int,
IfxType.BIGINT: _decode_bigint, IfxType.BIGINT: _decode_bigint,
IfxType.BIGSERIAL: _decode_bigint, IfxType.BIGSERIAL: _decode_bigint,
IfxType.INT8: _decode_int8,
IfxType.SERIAL8: _decode_int8,
IfxType.SMFLOAT: _decode_smfloat, IfxType.SMFLOAT: _decode_smfloat,
IfxType.FLOAT: _decode_float, IfxType.FLOAT: _decode_float,
IfxType.CHAR: _decode_char, IfxType.CHAR: _decode_char,
@ -700,8 +761,28 @@ def _encode_float(value: float) -> EncodedParam:
def _encode_bool(value: bool) -> EncodedParam: def _encode_bool(value: bool) -> EncodedParam:
"""Encode a Python bool as Informix BOOLEAN (type=45, 1 byte).""" """Encode a Python bool by binding the literal ``'t'`` / ``'f'``.
return (45, 0, b"\x01" if value else b"\x00")
Informix BOOLEAN has no bindable binary form we could find. This used
to send ``(45, 0, b"\\x01")``, and type 45 is not something the server
accepts as a bind type it simply stopped responding, so any
``execute`` with a bool parameter **hung until the read timeout**, or
forever if none was set. Type 41 (which is how BOOLEAN is *described*
in results) hangs the same way; the descriptor type and the bind type
are not interchangeable.
Sending ``'t'`` / ``'f'`` as CHAR and letting the server cast on
assignment is what works, and it mirrors Informix's own literal
syntax (``INSERT ... VALUES ('t')``). It also matches how
:func:`_encode_str` already leans on server-side conversion rather
than trying to match the destination column's exact type.
Note the asymmetry with the read path, which is correct and stays
as it is: BOOLEAN comes *back* as UDTFIXED (41) wrapped in a UDT
envelope whose payload byte is ``0x74``/``0x66`` ASCII ``t``/``f``,
the same characters we send here.
"""
return _encode_str("t" if value else "f")
def _encode_date(value: datetime.date) -> EncodedParam: def _encode_date(value: datetime.date) -> EncodedParam:
@ -717,9 +798,22 @@ def _encode_date(value: datetime.date) -> EncodedParam:
def _encode_datetime(value: datetime.datetime) -> EncodedParam: def _encode_datetime(value: datetime.datetime) -> EncodedParam:
"""Encode a Python ``datetime.datetime`` as Informix DATETIME (type=10). """Encode a Python ``datetime.datetime`` as Informix DATETIME (type=10).
Emit YEAR TO SECOND form covers the common case of stored Emits YEAR TO SECOND when ``microsecond`` is zero, and YEAR TO
timestamps without microseconds. (Phase 6.x can add YEAR TO FRACTION(5) when it isn't.
FRACTION(N) variants if microseconds are needed.)
The conditional matters. Emitting YEAR TO SECOND unconditionally
silently discarded sub-second precision on every bind: a
``DATETIME YEAR TO FRACTION(5)`` column handed
``datetime(..., microsecond=120000)`` stored ``.00000``, with no
error. Only widening when there is a fraction to carry keeps the
long-exercised YEAR TO SECOND path byte-identical for the common
case, and Informix converts between qualifiers on assignment, so a
FRACTION(5) bind into a narrower column truncates server-side
rather than failing.
FRACTION(5) is the widest Informix supports and holds 10 µs
resolution; Python's microsecond field is finer, so the last digit
is dropped. Same trade-off ``_encode_timedelta`` already makes.
Format (per ``Decimal.javaToIfx`` line 457): Format (per ``Decimal.javaToIfx`` line 457):
byte[0..1] = short total length of data following (= digit_count/2 + 1) byte[0..1] = short total length of data following (= digit_count/2 + 1)
@ -743,10 +837,20 @@ def _encode_datetime(value: datetime.datetime) -> EncodedParam:
(value.second, 2), (value.second, 2),
] ]
digit_str = "".join(f"{v:0{w}d}" for v, w in fields) # 14 digits digit_str = "".join(f"{v:0{w}d}" for v, w in fields) # 14 digits
if value.microsecond:
# 6 fraction digits = exactly 3 more BCD pairs, so the digit
# string stays even and the exponent byte is unchanged (the
# integer part is still 7 base-100 pairs). FRACTION(5) carries
# 5 significant digits; the 6th is padding the wire format
# requires. Qualifier: digit_count=19, start=YEAR(0), end=
# FRACTION(5)=15 — matching what the decoder reads back.
digit_str += f"{value.microsecond:06d}" # -> 20 digits
prec = (19 << 8) | (0 << 4) | 15
else:
prec = (14 << 8) | (0 << 4) | 10
digit_bytes = bytes(int(digit_str[i : i + 2]) for i in range(0, len(digit_str), 2)) digit_bytes = bytes(int(digit_str[i : i + 2]) for i in range(0, len(digit_str), 2))
inner = bytes([0xC7]) + digit_bytes # 8 bytes (1 exp + 7 BCD pairs) inner = bytes([0xC7]) + digit_bytes # 1 exp byte + 7 or 10 BCD pairs
raw = len(inner).to_bytes(2, "big") + inner # +2 byte length prefix = 10 bytes raw = len(inner).to_bytes(2, "big") + inner # +2 byte length prefix
prec = (14 << 8) | (0 << 4) | 10
return (10, prec, raw) return (10, prec, raw)

View File

@ -22,6 +22,7 @@ from __future__ import annotations
import contextlib import contextlib
import itertools import itertools
import logging import logging
import os
import struct import struct
import weakref import weakref
from collections.abc import Iterator from collections.abc import Iterator
@ -29,10 +30,16 @@ from typing import TYPE_CHECKING, Any
from . import _errcodes from . import _errcodes
from ._messages import MessageType from ._messages import MessageType
from ._protocol import IfxStreamReader, make_pdu_writer from ._protocol import (
WIRE_ERRORS,
BufferedSocketReader,
IfxStreamReader,
make_pdu_writer,
)
from ._resultset import ( from ._resultset import (
ColumnInfo, ColumnInfo,
compile_column_readers, compile_column_readers,
compile_row_decoder,
parse_describe, parse_describe,
parse_tuple_payload, parse_tuple_payload,
) )
@ -43,6 +50,7 @@ from .exceptions import (
NotSupportedError, NotSupportedError,
ProgrammingError, ProgrammingError,
) )
from .rows import Row, make_row_class
if TYPE_CHECKING: if TYPE_CHECKING:
from .connections import Connection from .connections import Connection
@ -53,7 +61,6 @@ if TYPE_CHECKING:
_cursor_counter = itertools.count(1) _cursor_counter = itertools.count(1)
_NUMERIC_PLACEHOLDER_RE = __import__("re").compile(r":(\d+)")
# Phase 28: pre-built CLOSE and RELEASE PDU bytes for cursor finalizers. # Phase 28: pre-built CLOSE and RELEASE PDU bytes for cursor finalizers.
@ -76,6 +83,67 @@ del _build_static_pdu
_log = logging.getLogger(__name__) _log = logging.getLogger(__name__)
# Phase 39: buffered socket reader is the default. The reader holds a
# connection-scoped bytearray + offset cursor on ``IfxSocket``, so one
# ``recv()`` per ~64 KB amortizes over hundreds of fields rather than
# the per-field-recv pattern the legacy ``_SocketReader`` uses.
#
# Set ``IFX_BUFFERED_READER=0`` to fall back to the legacy reader if
# you hit a wire shape we haven't covered. The legacy path is kept as
# the escape hatch (and as the A/B baseline if we ever need to bisect
# a regression). Read once at module import so the check isn't on the
# hot path.
_USE_BUFFERED_READER = os.environ.get("IFX_BUFFERED_READER", "1") != "0"
def _make_socket_reader(sock):
if _USE_BUFFERED_READER:
return BufferedSocketReader(sock)
return _SocketReader(sock)
# Statement types from the DESCRIBE response's first field. The server
# tells us exactly what it prepared; these are the two values that mean
# "this will produce rows".
_ST_SELECT = 2
_ST_ROUTINE = 56 # EXECUTE PROCEDURE / EXECUTE FUNCTION
# Transaction control run as SQL. JDBC reads the same three values off
# the describe and calls setTxBeginState / setTxEndState (IfxSqli, the
# TxStmt field). Both ``BEGIN`` and ``BEGIN WORK`` report 34; likewise
# the WORK-less spellings of the other two.
_ST_TX_BEGIN = 34
_ST_TX_COMMIT = 35
_ST_TX_ROLLBACK = 36
_TX_CONTROL_TYPES = frozenset({_ST_TX_BEGIN, _ST_TX_COMMIT, _ST_TX_ROLLBACK})
def _produces_result_set(statement_type: int, ncolumns: int) -> bool:
"""Whether a prepared statement needs a cursor opened for it.
This is JDBC's ``IfxSqli.isResultSet`` predicate, and it replaces a
first-word check for ``SELECT`` that got five ordinary forms wrong.
A leading comment of any of the three Informix flavours, a
parenthesized select, a parenthesized UNION, and a CTE were all
classified as DML, so the driver sent SQ_EXECUTE where the server
expected a cursor. The report back was ``-260 Cursor name already in
use``, which describes neither the cause nor anything the caller did.
The original comment justified the heuristic on the grounds that
``nfields`` can't distinguish these, because ``INSERT INTO t VALUES
(?)`` also describes a column. True, and irrelevant: ``statement_type``
distinguishes them exactly. That INSERT reports 6 with one field; every
SELECT form above reports 2. The value was being parsed into the
describe metadata and thrown away.
``EXECUTE PROCEDURE``/``FUNCTION`` (56) is the one type that depends on
the column count, since a routine may or may not return rows.
"""
if statement_type == _ST_SELECT:
return True
return statement_type == _ST_ROUTINE and ncolumns > 0
def _finalize_cursor( def _finalize_cursor(
conn_ref: weakref.ReferenceType, conn_ref: weakref.ReferenceType,
state: list, state: list,
@ -99,13 +167,30 @@ def _finalize_cursor(
a list (not the cursor object itself) keeps the finalizer's closure a list (not the cursor object itself) keeps the finalizer's closure
weak the cursor remains GC'd-able. weak the cursor remains GC'd-able.
""" """
from ._protocol import ProtocolError
if not state[0]: if not state[0]:
return # nothing to release return # nothing to release
conn = conn_ref() conn = conn_ref()
if conn is None or conn.closed: if conn is None or conn.closed:
return return
if conn._wire_lock.held_by_current_thread:
# GC fired on the very thread that is mid-statement. The
# non-blocking acquire below would *succeed* here — an RLock
# grants a reentrant acquire to its own owner — and we would
# send CLOSE/RELEASE into the middle of that statement, killing
# it with -208. Defer instead, exactly as for another thread.
#
# Refcounting hides this: a dropped cursor is freed at the drop,
# before the next statement. A cursor caught in a reference cycle
# waits for a collection instead, and cycles are ordinary — any
# traceback holding a cursor makes one.
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
_log.debug(
"cursor finalizer: GC ran on the thread holding the wire lock; "
"enqueued CLOSE+RELEASE for deferred cleanup on conn %s",
id(conn),
)
return
if not conn._wire_lock.acquire(blocking=False): if not conn._wire_lock.acquire(blocking=False):
# Another thread is mid-operation on this connection. Don't # Another thread is mid-operation on this connection. Don't
# deadlock; instead, hand the cleanup bytes to the connection's # deadlock; instead, hand the cleanup bytes to the connection's
@ -126,19 +211,33 @@ def _finalize_cursor(
conn._drain_to_eot() conn._drain_to_eot()
conn._send_pdu(_RELEASE_PDU) conn._send_pdu(_RELEASE_PDU)
conn._drain_to_eot() conn._drain_to_eot()
except (ProtocolError, OSError) as exc: except WIRE_ERRORS as exc:
# Wire desync during cleanup — same doctrine as if getattr(exc, "sqlcode", None) is not None:
# ``_raise_sq_err``: the wire is unrecoverable, force-close # The *server* rejected the cleanup — typically a stale
# the connection. Asymmetric handling of the same failure # CLOSE for a cursor it no longer has, which answers
# mode would be a Hamilton smell. # -267 "the transaction has been rolled back". That is
_log.warning( # an OperationalError, which is in WIRE_ERRORS, so this
"cursor finalizer: wire desync during cleanup; " # branch used to force-close a perfectly healthy
"force-closing connection: %r", # connection over a no-op. ``_raise_sq_err`` self-drains
exc, # the trailing SQ_EOT, so the wire is still aligned.
) # Same distinction as _drain_pending_cleanup.
conn._closed = True _log.debug(
with contextlib.suppress(Exception): "cursor finalizer: server rejected cleanup (stale): %r",
conn._sock.close() exc,
)
else:
# No sqlcode: the socket died or framing desynced and we
# can no longer say where a response ends. Force-close —
# same doctrine as ``_raise_sq_err``. Asymmetric handling
# of the same failure mode would be a Hamilton smell.
_log.warning(
"cursor finalizer: wire desync during cleanup; "
"force-closing connection: %r",
exc,
)
conn._closed = True
with contextlib.suppress(Exception):
conn._sock.close()
except InterfaceError: except InterfaceError:
# Connection was closed by another thread between our # Connection was closed by another thread between our
# ``conn.closed`` check above and the actual write. No-op: # ``conn.closed`` check above and the actual write. No-op:
@ -148,17 +247,103 @@ def _finalize_cursor(
conn._wire_lock.release() conn._wire_lock.release()
_ASCII_DIGITS = frozenset("0123456789")
def _rewrite_numeric_to_qmark(sql: str) -> str: def _rewrite_numeric_to_qmark(sql: str) -> str:
"""Convert ``:1`` / ``:2`` placeholders (paramstyle="numeric") to ``?``. """Convert ``:1`` / ``:2`` placeholders (paramstyle="numeric") to ``?``.
Informix's wire protocol uses ``?`` natively. Since we expose Informix's wire protocol uses ``?`` natively, and we advertise
``paramstyle="numeric"`` in the public API (matches Informix ``paramstyle="numeric"`` to match the ESQL/C convention, so the
ESQL/C convention), we rewrite before sending. Trivial cases only placeholders are rewritten on the way out.
strings and comments are NOT escaped, so SQL containing literal
``:1`` inside string literals will be wrongly substituted. Phase 5 This used to be ``re.sub(r":(\\d+)", "?", sql)``, which cannot see a
can add a proper SQL tokenizer. string literal and so rewrote the contents of one. Any ``HH:MM``
time, any URL with a port, any aspect ratio::
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
stored as 'http://host?/x'
Nothing about that is opt-in. The rewrite runs whenever a statement
has parameters, whatever placeholder style the caller actually used,
so writing ``?`` everywhere and never touching numeric style did not
protect you. It also changed the placeholder *count* while
``num_qmarks`` was still computed from ``len(params)``, leaving the
driver and the server disagreeing about how many binds exist.
So: a single pass that substitutes only outside quotes and comments.
The lexical rules are Informix's own, measured against 12.10, 14.10
and 15 rather than assumed from standard SQL:
* ``''`` doubling escapes a quote inside ``'...'``. A backslash does
**not** escape anything; ``'a\'b'`` is an unterminated string and
the server answers ``-282``. A Postgres-style scanner that honours
``\'`` would desync here and corrupt everything after it.
* ``"..."`` is a delimited identifier or a string depending on
``DELIMIDENT``. Either way its contents are not ours to touch.
* ``--`` runs to end of line, ``/* */`` does **not** nest (the first
``*/`` closes it; nesting is a syntax error), and ``{ }`` is a
comment too.
* ``::`` is the cast operator and is stepped over as a unit, so it
can never be read as the start of a placeholder.
An unterminated quote or comment consumes the rest of the string and
substitutes nothing further. That is deliberate: under-substituting
leaves the server to reject SQL that was already malformed, while
guessing would corrupt a literal.
""" """
return _NUMERIC_PLACEHOLDER_RE.sub("?", sql) if ":" not in sql:
return sql
out: list[str] = []
i = 0
n = len(sql)
while i < n:
ch = sql[i]
if ch in ("'", '"'):
j = i + 1
while j < n:
if sql[j] == ch:
if j + 1 < n and sql[j + 1] == ch:
j += 2 # doubled quote, still inside
continue
j += 1
break
j += 1
out.append(sql[i:j])
i = j
elif ch == "-" and sql.startswith("--", i):
j = sql.find("\n", i)
j = n if j == -1 else j
out.append(sql[i:j])
i = j
elif ch == "/" and sql.startswith("/*", i):
j = sql.find("*/", i + 2)
j = n if j == -1 else j + 2
out.append(sql[i:j])
i = j
elif ch == "{":
j = sql.find("}", i)
j = n if j == -1 else j + 1
out.append(sql[i:j])
i = j
elif ch == ":":
if sql.startswith("::", i):
out.append("::")
i += 2
continue
j = i + 1
while j < n and sql[j] in _ASCII_DIGITS:
j += 1
if j > i + 1:
out.append("?")
i = j
else:
out.append(ch)
i += 1
else:
out.append(ch)
i += 1
return "".join(out)
def _generate_cursor_name() -> str: def _generate_cursor_name() -> str:
@ -189,9 +374,13 @@ class Cursor:
# manipulation. Two-mode cursor; the same surface API works # manipulation. Two-mode cursor; the same surface API works
# for both. # for both.
self._scrollable = scrollable self._scrollable = scrollable
# Inherited from the connection, overridable per cursor. See
# informix_db.rows for what this costs and why it is opt-in.
self.row_factory = connection.row_factory
self._description: list[tuple] | None = None self._description: list[tuple] | None = None
self._columns: list[ColumnInfo] = [] self._columns: list[ColumnInfo] = []
self._column_readers: list[tuple] | None = None # Phase 37 self._column_readers: list[tuple] | None = None # Phase 37
self._row_decoder = None # Phase 38 codegen'd row decoder
self._rowcount: int = -1 self._rowcount: int = -1
self._rows: list[tuple] = [] self._rows: list[tuple] = []
# Phase 17: index-based row access enables scroll cursors. The # Phase 17: index-based row access enables scroll cursors. The
@ -225,6 +414,11 @@ class Cursor:
# from. Empirically the server accepts 0 here even when a real # from. Empirically the server accepts 0 here even when a real
# ID was assigned, so this is best-effort tracking. # ID was assigned, so this is best-effort tracking.
self._statement_id: int = 0 self._statement_id: int = 0
# DESCRIBE's statement-type field. Decides whether a cursor is
# opened -- see _produces_result_set.
self._statement_type: int = 0
# Per-result-set row class when row_factory is set, else None.
self._row_class: type | None = None
# Phase 10: smart-LOB read via ``lotofile(col, path, 'client')``. # Phase 10: smart-LOB read via ``lotofile(col, path, 'client')``.
# The server orchestrates a SQ_FILE (98) protocol where it tells # The server orchestrates a SQ_FILE (98) protocol where it tells
# us to "open file X, write these bytes, close". We emulate the # us to "open file X, write these bytes, close". We emulate the
@ -309,34 +503,54 @@ class Cursor:
def _execute_under_wire_lock(self, sql: str, params: tuple) -> None: def _execute_under_wire_lock(self, sql: str, params: tuple) -> None:
"""Wire-bound body of ``execute``. Caller MUST hold ``_wire_lock``.""" """Wire-bound body of ``execute``. Caller MUST hold ``_wire_lock``."""
# Someone else's scrollable cursor owns the session's statement
# slot -- refuse rather than draw a -285 and destroy their cursor.
self._conn._check_scroll_cursor_conflict(self)
# Our own previous scrollable cursor is still open server-side.
# Re-executing over the top of it collides exactly the same way,
# so close it first. This is the one caller allowed past the
# check above, and it is only safe because of this.
if self._scrollable and self._server_cursor_open:
self._close_server_cursor()
# Reset previous-execute state. # Reset previous-execute state.
self._description = None self._description = None
self._columns = [] self._columns = []
self._column_readers = None # Phase 37 self._column_readers = None # Phase 37
self._row_decoder = None # Phase 38
self._rowcount = -1 self._rowcount = -1
self._rows = [] self._rows = []
self._row_index = -1 # before-first-row self._row_index = -1 # before-first-row
self._row_class = None
self._statement_type = 0
self._statement_already_done = False self._statement_already_done = False
# Step 1: PREPARE — send SQL with numQmarks = len(params).
# statement_boundary: nothing is open server-side yet, so this is
# the one safe moment to flush a finalizer's deferred cleanup.
self._conn._send_pdu(
self._build_prepare_pdu(sql, num_qmarks=len(params)),
statement_boundary=True,
)
self._read_describe_response()
# On a logged DB in non-autocommit mode, the server requires an # On a logged DB in non-autocommit mode, the server requires an
# explicit SQ_BEGIN before the first DML in each transaction. # explicit SQ_BEGIN before the first DML in each transaction.
# _ensure_transaction is a no-op for autocommit / unlogged DBs, # _ensure_transaction is a no-op for autocommit / unlogged DBs,
# and idempotent within an open transaction. # and idempotent within an open transaction.
self._conn._ensure_transaction() #
# This runs *after* the describe, matching JDBC's
# initiateTransaction placement, because until the describe lands
# we don't know whether the caller's statement is itself
# transaction control. Opening a transaction on their behalf and
# then executing their BEGIN WORK gets -535, "already in
# transaction" — the driver competing with the user for the same
# job and the user losing.
if self._statement_type not in _TX_CONTROL_TYPES:
self._conn._ensure_transaction()
# Step 1: PREPARE — send SQL with numQmarks = len(params). # Ask the server what it just prepared, rather than guessing from
self._conn._send_pdu(self._build_prepare_pdu(sql, num_qmarks=len(params))) # the first word of the SQL.
self._read_describe_response() if _produces_result_set(self._statement_type, len(self._columns)):
# Branch on the SQL keyword. We can't use ``self._columns`` /
# ``nfields`` here because a parameterized INSERT also returns
# a non-empty DESCRIBE (server describes the would-be inserted
# row's columns). The SQL-keyword heuristic is what JDBC effectively
# does too via its IfxStatement / IfxPreparedStatement subclassing.
first_word = sql.lstrip().split(None, 1)[0].upper() if sql.strip() else ""
is_select = first_word == "SELECT"
if is_select:
if params: if params:
self._execute_select_with_params(params) self._execute_select_with_params(params)
else: else:
@ -346,6 +560,13 @@ class Cursor:
else: else:
self._execute_dml() self._execute_dml()
self._row_class = self._resolve_row_class()
# The statement succeeded. If it was transaction control, the
# server's transaction state just changed and the connection has
# to know, or commit() and rollback() silently do nothing.
self._note_transaction_control()
# SELECT path: position cursor before the first row so the next # SELECT path: position cursor before the first row so the next
# ``fetchone()`` returns ``rows[0]``. DML paths leave _row_index # ``fetchone()`` returns ``rows[0]``. DML paths leave _row_index
# at -1 too (no rows to iterate). # at -1 too (no rows to iterate).
@ -354,6 +575,98 @@ class Cursor:
if self._description is not None: if self._description is not None:
self._row_index = -1 self._row_index = -1
def _resolve_row_class(self) -> type | None:
"""Pick the class this result set's rows are handed back as.
``row_factory`` is either :class:`informix_db.Row` (or a subclass),
in which case the per-shape class is built and cached from the
column names, or any callable taking the name tuple and returning
something that takes a values tuple.
Returns ``None`` when no factory is set, which is the default and
keeps plain tuples on the hot path at zero cost.
"""
factory = self.row_factory
if factory is None or self._description is None:
return None
names = tuple(d[0] for d in self._description)
if isinstance(factory, type) and issubclass(factory, Row):
return make_row_class(names)
return factory(names)
def _note_transaction_control(self) -> None:
"""Sync the connection's transaction flag after a successful execute.
``BEGIN WORK`` run through ``execute()`` opens a real transaction
on the server, and the connection had no idea. With autocommit on
nothing stopped it reaching the server, so ``_in_transaction``
stayed False while a transaction was open and both ``commit()``
and ``rollback()`` are guarded by that flag. ``rollback()``
returned successfully having sent nothing, and the rows it was
asked to discard were still there.
The pool reads the same flag to decide whether a returned
connection needs cleaning up, so the connection went back into
circulation holding an open transaction and its locks.
Called only on the success path: a statement that failed did not
change the server's transaction state.
"""
statement_type = self._statement_type
if statement_type == _ST_TX_BEGIN:
self._conn._in_transaction = True
elif statement_type in (_ST_TX_COMMIT, _ST_TX_ROLLBACK):
self._conn._in_transaction = False
def _close_server_cursor(self) -> None:
"""Free the server-side scrollable cursor. Caller MUST hold ``_wire_lock``.
Best-effort: a wire failure here is swallowed. Both callers are
already past the point where reporting it would help one is
closing the cursor, the other is about to run a new statement
that will report its own failure if the wire is really gone.
"""
try:
self._conn._send_pdu(self._build_close_pdu())
self._drain_to_eot()
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
except Exception:
pass
self._server_cursor_open = False
self._conn._open_scroll_cursor = None
def _release_after_failure(self, *, close_cursor: bool = False) -> None:
"""Best-effort server-side cleanup after a statement fails.
A statement that failed is still allocated. Skipping the release
bricks the connection: the next PREPARE collides with the leaked
one and every subsequent call returns a nonsense error whose
offset points back at the *failed* SQL. A duplicate-key violation
is about the most ordinary error an application can hit, so "one
constraint violation kills the connection" was easy to reach and
hard to attribute.
CLOSE and RELEASE get **separate** suppressions. Putting both in
one ``with contextlib.suppress(Exception)`` block reads as "clean
up both", but a CLOSE that raises skips the RELEASE entirely —
and RELEASE is the one that matters. Failing to close a cursor
wastes a handle; failing to release the statement is what breaks
the next call.
Everything here is swallowed rather than propagated. If the wire
is genuinely desynced then the cleanup fails too, and the caller
is far better served by the real SQL error than by a secondary
failure from the cleanup path.
"""
if close_cursor:
with contextlib.suppress(Exception):
self._conn._send_pdu(self._build_close_pdu())
self._drain_to_eot()
with contextlib.suppress(Exception):
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
def _execute_select_with_params(self, params: tuple) -> None: def _execute_select_with_params(self, params: tuple) -> None:
"""Parameterized SELECT: SQ_BIND → CURNAME+NFETCH → drain → CLOSE+RELEASE. """Parameterized SELECT: SQ_BIND → CURNAME+NFETCH → drain → CLOSE+RELEASE.
@ -372,12 +685,20 @@ class Cursor:
try: try:
pdu = self._build_bind_only_pdu(params) pdu = self._build_bind_only_pdu(params)
except Exception: except Exception:
with contextlib.suppress(Exception): self._release_after_failure()
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
raise raise
self._conn._send_pdu(pdu) self._conn._send_pdu(pdu)
self._drain_to_eot() try:
self._drain_to_eot()
except Exception:
# The server can reject the BIND itself — a value whose type
# doesn't match the described parameter, a bind against a
# statement the server has since invalidated. This was the
# last unguarded door of the six: the build was covered and
# the drain was not, so a server-side bind rejection left the
# statement allocated and the *next* execute() failed instead.
self._release_after_failure()
raise
# Now open the cursor and fetch — the bound values are in scope # Now open the cursor and fetch — the bound values are in scope
# for the prepared statement. # for the prepared statement.
self._execute_select() self._execute_select()
@ -400,9 +721,23 @@ class Cursor:
self._conn._send_pdu( self._conn._send_pdu(
self._build_curname_scroll_open_pdu(cursor_name) self._build_curname_scroll_open_pdu(cursor_name)
) )
self._drain_to_eot() try:
self._drain_to_eot()
except Exception:
# Opening a scrollable cursor fails like any other
# statement — a bad ORDER BY, a permission error, a table
# dropped between PREPARE and OPEN. This branch had no
# guard at all, and it is the worst place to lack one:
# the GC-time finalizer is armed on the line *after* the
# drain, so a failure here left the statement allocated
# with no fallback whatsoever to reclaim it.
self._release_after_failure(close_cursor=True)
raise
self._server_cursor_open = True self._server_cursor_open = True
self._finalizer_state[0] = True # arm the GC-time fallback self._finalizer_state[0] = True # arm the GC-time fallback
# The connection needs to know its statement slot is taken,
# so the next statement can refuse instead of drawing -285.
self._conn._open_scroll_cursor = weakref.ref(self)
self._scroll_total_rows = None self._scroll_total_rows = None
return # don't close; cursor stays live for SQ_SFETCH return # don't close; cursor stays live for SQ_SFETCH
# Phase 35: NFETCH loop — keep fetching until a response yields # Phase 35: NFETCH loop — keep fetching until a response yields
@ -412,20 +747,32 @@ class Cursor:
# This bug was latent for ~30 phases because no test used a # This bug was latent for ~30 phases because no test used a
# large enough result set to trigger it. # large enough result set to trigger it.
self._conn._send_pdu(self._build_curname_nfetch_pdu(cursor_name)) self._conn._send_pdu(self._build_curname_nfetch_pdu(cursor_name))
rows_before = len(self._rows) try:
self._read_fetch_response()
rows_received = len(self._rows) - rows_before
while rows_received > 0:
self._conn._send_pdu(self._build_nfetch_pdu())
rows_before = len(self._rows) rows_before = len(self._rows)
self._read_fetch_response() self._read_fetch_response()
rows_received = len(self._rows) - rows_before rows_received = len(self._rows) - rows_before
# Dereference BYTE/TEXT blob descriptors BEFORE CLOSE — the while rows_received > 0:
# locators are only valid while the cursor is open. No-op when self._conn._send_pdu(self._build_nfetch_pdu())
# no BYTE/TEXT columns are present. rows_before = len(self._rows)
self._dereference_blob_columns() self._read_fetch_response()
rows_received = len(self._rows) - rows_before
# Dereference BYTE/TEXT blob descriptors BEFORE CLOSE — the
# locators are only valid while the cursor is open. No-op when
# no BYTE/TEXT columns are present.
self._dereference_blob_columns()
except Exception:
# A raise anywhere in the fetch loop leaves the cursor and the
# statement allocated server-side. The GC-time finalizer only
# covers scrollable cursors (it is armed above, in the branch
# that returns early), so this path needs its own cleanup —
# otherwise a mid-fetch failure leaks and the next statement
# collides with it. Same failure mode as the DML path; see
# _release_after_failure for what that looks like from the
# caller's side.
self._release_after_failure(close_cursor=True)
raise
self._conn._send_pdu(self._build_close_pdu()) self._conn._send_pdu(self._build_close_pdu())
self._drain_to_eot() self._drain_to_eot()
@ -767,7 +1114,7 @@ class Cursor:
writer.write_short(MessageType.SQ_EOT) writer.write_short(MessageType.SQ_EOT)
self._conn._send_pdu(buf.getvalue()) self._conn._send_pdu(buf.getvalue())
reader = _SocketReader(self._conn._sock) reader = _make_socket_reader(self._conn._sock)
chunks: list[bytes] = [] chunks: list[bytes] = []
while True: while True:
tag = reader.read_short() tag = reader.read_short()
@ -811,12 +1158,17 @@ class Cursor:
try: try:
pdu = self._build_bind_execute_pdu(params) pdu = self._build_bind_execute_pdu(params)
except Exception: except Exception:
with contextlib.suppress(Exception): self._release_after_failure()
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
raise raise
self._conn._send_pdu(pdu) self._conn._send_pdu(pdu)
self._drain_to_eot() try:
self._drain_to_eot()
except Exception:
# The statement is still allocated server-side even though it
# failed. See _execute_dml for why skipping this bricks the
# connection.
self._release_after_failure()
raise
self._conn._send_pdu(self._build_release_pdu()) self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot() self._drain_to_eot()
@ -836,7 +1188,27 @@ class Cursor:
let the optimization-looking response confuse you. let the optimization-looking response confuse you.
""" """
self._conn._send_pdu(self._build_execute_pdu()) self._conn._send_pdu(self._build_execute_pdu())
self._drain_to_eot() # reads DONE + COST + EOT, populates rowcount try:
self._drain_to_eot() # reads DONE + COST + EOT, populates rowcount
except Exception:
# A statement that FAILS is still allocated on the server, and
# skipping the release here bricked the whole connection: the
# next PREPARE collided with the leaked statement and every
# subsequent call returned a nonsense error (-255 "Not in
# transaction" in autocommit, -285 otherwise) whose offset
# pointed back at the *failed* SQL, not the new statement.
#
# A duplicate-key violation is about the most ordinary error an
# application can hit, so "one constraint violation kills the
# connection" was easy to reach and hard to attribute. Send the
# release, then re-raise the original error.
#
# Suppressed rather than propagated: if the wire is genuinely
# desynced the release will fail too, and the caller is far
# better served by the real SQL error than by a secondary
# failure from the cleanup path.
self._release_after_failure()
raise
self._conn._send_pdu(self._build_release_pdu()) self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot() self._drain_to_eot()
@ -893,7 +1265,10 @@ class Cursor:
f"expected {first_len} (matching set [0])" f"expected {first_len} (matching set [0])"
) )
# Detect SELECT — not supported in executemany. # Cheap pre-flight reject for the obvious case, so the common
# mistake costs no round-trip. The authoritative check is after
# PREPARE, below — this one shares the first-word heuristic's
# blind spots (leading comments, CTEs, parenthesized selects).
first_word = operation.lstrip().split(None, 1)[0].upper() if operation.strip() else "" first_word = operation.lstrip().split(None, 1)[0].upper() if operation.strip() else ""
if first_word == "SELECT": if first_word == "SELECT":
raise NotSupportedError("executemany on SELECT is not supported") raise NotSupportedError("executemany on SELECT is not supported")
@ -904,6 +1279,9 @@ class Cursor:
# under the wire lock — N rows commit atomically with respect # under the wire lock — N rows commit atomically with respect
# to other threads on the connection. # to other threads on the connection.
with self._conn._wire_lock: with self._conn._wire_lock:
self._conn._check_scroll_cursor_conflict(self)
if self._scrollable and self._server_cursor_open:
self._close_server_cursor()
# Reset per-execute state. # Reset per-execute state.
self._description = None self._description = None
self._columns = [] self._columns = []
@ -911,27 +1289,61 @@ class Cursor:
self._rowcount = -1 self._rowcount = -1
self._rows = [] self._rows = []
self._row_index = -1 self._row_index = -1
self._statement_type = 0
self._statement_already_done = False self._statement_already_done = False
# Logged-DB transaction guard — same as execute(). Idempotent
# within an open transaction.
self._conn._ensure_transaction()
# PREPARE once. # PREPARE once.
self._conn._send_pdu( self._conn._send_pdu(
self._build_prepare_pdu(sql, num_qmarks=first_len) self._build_prepare_pdu(sql, num_qmarks=first_len),
statement_boundary=True,
) )
self._read_describe_response() self._read_describe_response()
# Now the server has told us what it prepared. A result-set
# statement that slipped past the first-word check above --
# a CTE, a leading comment, a parenthesized select -- would
# otherwise be executed N times down the DML path, which
# opens no cursor and answers -260.
if _produces_result_set(self._statement_type, len(self._columns)):
self._release_after_failure()
raise NotSupportedError(
"executemany on a statement that returns rows is not "
"supported"
)
# Logged-DB transaction guard — same as execute(), and for the
# same reason placed after the describe rather than before it.
if self._statement_type not in _TX_CONTROL_TYPES:
self._conn._ensure_transaction()
# Phase 33: pipeline — build all BIND+EXECUTE PDUs first # Phase 33: pipeline — build all BIND+EXECUTE PDUs first
# (Python work, no I/O), then send them back-to-back, then # (Python work, no I/O), then send them back-to-back, then
# drain all responses. Eliminates the per-row round-trip # drain all responses. Eliminates the per-row round-trip
# the older serial loop paid. # the older serial loop paid.
pdus = [ try:
self._build_bind_execute_pdu(tuple(p)) for p in seq pdus = [
] self._build_bind_execute_pdu(tuple(p)) for p in seq
for pdu in pdus: ]
self._conn._send_pdu(pdu) for pdu in pdus:
self._conn._send_pdu(pdu)
except Exception:
# Encoding a row can fail client-side (DataError for a
# value the connection's codec can't represent), and it
# fails here — after the PREPARE, before anything is
# drained. Escaping without the RELEASE leaks the
# statement server-side, and the next PREPARE then
# collides with it: every later call on this connection
# returns a nonsense error pointing at the *previous*
# SQL. ``_execute_dml_with_params`` guards the identical
# case for the single-row path; the pipelined path was
# simply missed.
#
# A send failure lands here too. The wire is likely
# already unusable in that case, but attempting the
# release costs nothing and the original error is what
# propagates either way.
self._release_after_failure()
raise
# Drain N responses. The first error is captured but we # Drain N responses. The first error is captured but we
# still drain the rest (they're SQ_ERRs for the aborted # still drain the rest (they're SQ_ERRs for the aborted
@ -1003,7 +1415,8 @@ class Cursor:
self._row_index = len(self._rows) # past-last self._row_index = len(self._rows) # past-last
return None return None
self._row_index = nxt self._row_index = nxt
return self._rows[nxt] row = self._rows[nxt]
return self._row_class(row) if self._row_class is not None else row
def fetchmany(self, size: int | None = None) -> list[tuple]: def fetchmany(self, size: int | None = None) -> list[tuple]:
self._check_open() self._check_open()
@ -1034,6 +1447,8 @@ class Cursor:
return [] return []
start = self._row_index + 1 start = self._row_index + 1
out = self._rows[start:] out = self._rows[start:]
if self._row_class is not None:
out = [self._row_class(r) for r in out]
self._row_index = len(self._rows) self._row_index = len(self._rows)
return list(out) return list(out)
@ -1214,7 +1629,7 @@ class Cursor:
if scrolltype == 4 or is_last_probe: if scrolltype == 4 or is_last_probe:
# SFETCH(LAST) — TUPID == total row count # SFETCH(LAST) — TUPID == total row count
self._scroll_total_rows = self._last_tupid self._scroll_total_rows = self._last_tupid
return row return self._row_class(row) if self._row_class is not None else row
def close(self) -> None: def close(self) -> None:
"""Close the cursor. """Close the cursor.
@ -1226,20 +1641,8 @@ class Cursor:
if self._closed: if self._closed:
return return
if self._scrollable and self._server_cursor_open: if self._scrollable and self._server_cursor_open:
# Phase 27: hold the wire lock during CLOSE+RELEASE so we with self._conn._wire_lock:
# don't interleave with another thread's pending op on the self._close_server_cursor()
# connection. Best-effort: any wire failure here is
# swallowed (the caller is closing; we don't want to mask
# whatever caused them to close).
try:
with self._conn._wire_lock:
self._conn._send_pdu(self._build_close_pdu())
self._drain_to_eot()
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
except Exception:
pass
self._server_cursor_open = False
# Phase 28: explicit close has handled the server-side resources # Phase 28: explicit close has handled the server-side resources
# (or tried to). Disarm the finalizer so it doesn't fire later # (or tried to). Disarm the finalizer so it doesn't fire later
# for nothing — and clear the state flag as a belt-and-suspenders # for nothing — and clear the state flag as a belt-and-suspenders
@ -1535,7 +1938,7 @@ class Cursor:
def _read_describe_response(self) -> None: def _read_describe_response(self) -> None:
"""Read DESCRIBE (+ optional SQ_INSERTDONE) + DONE + COST + EOT after PREPARE.""" """Read DESCRIBE (+ optional SQ_INSERTDONE) + DONE + COST + EOT after PREPARE."""
reader = _SocketReader(self._conn._sock) reader = _make_socket_reader(self._conn._sock)
while True: while True:
tag = reader.read_short() tag = reader.read_short()
if tag == MessageType.SQ_EOT: if tag == MessageType.SQ_EOT:
@ -1543,6 +1946,7 @@ class Cursor:
elif tag == MessageType.SQ_DESCRIBE: elif tag == MessageType.SQ_DESCRIBE:
self._columns, meta = parse_describe(reader) self._columns, meta = parse_describe(reader)
self._statement_id = meta.get("statement_id", 0) self._statement_id = meta.get("statement_id", 0)
self._statement_type = meta.get("statement_type", 0)
self._description = ( self._description = (
[c.to_description_tuple() for c in self._columns] if self._columns else None [c.to_description_tuple() for c in self._columns] if self._columns else None
) )
@ -1550,9 +1954,20 @@ class Cursor:
# row-decode loop in parse_tuple_payload uses this to avoid # row-decode loop in parse_tuple_payload uses this to avoid
# re-running per-row dispatch decisions that depend only # re-running per-row dispatch decisions that depend only
# on column metadata. # on column metadata.
self._column_readers = ( if self._columns:
compile_column_readers(self._columns) if self._columns else None self._column_readers = compile_column_readers(self._columns)
) # Phase 38: take it one step further — codegen a
# specialized row decoder for THIS column shape.
# Eliminates the per-column iteration overhead of
# the readers loop. ``None`` if codegen can't
# handle the shape; parse_tuple_payload then
# falls back to the readers-list dispatch.
self._row_decoder = compile_row_decoder(
self._column_readers, self._columns
)
else:
self._column_readers = None
self._row_decoder = None
elif tag == 94: # SQ_INSERTDONE — Informix optimization: literal elif tag == 94: # SQ_INSERTDONE — Informix optimization: literal
# INSERT executed during PREPARE. Payload is: # INSERT executed during PREPARE. Payload is:
# readLongInt (10 bytes) — serial8 inserted # readLongInt (10 bytes) — serial8 inserted
@ -1575,7 +1990,7 @@ class Cursor:
def _read_fetch_response(self) -> None: def _read_fetch_response(self) -> None:
"""Read TUPLE* + DONE + COST + EOT after an NFETCH or SFETCH.""" """Read TUPLE* + DONE + COST + EOT after an NFETCH or SFETCH."""
reader = _SocketReader(self._conn._sock) reader = _make_socket_reader(self._conn._sock)
while True: while True:
tag = reader.read_short() tag = reader.read_short()
if tag == MessageType.SQ_EOT: if tag == MessageType.SQ_EOT:
@ -1586,6 +2001,7 @@ class Cursor:
self._columns, self._columns,
encoding=self._conn.encoding, encoding=self._conn.encoding,
readers=self._column_readers, readers=self._column_readers,
row_decoder=self._row_decoder,
) )
self._rows.append(row) self._rows.append(row)
elif tag == MessageType.SQ_DONE: elif tag == MessageType.SQ_DONE:
@ -1608,7 +2024,7 @@ class Cursor:
def _drain_to_eot(self) -> None: def _drain_to_eot(self) -> None:
"""Read response stream until SQ_EOT, allowing common tags in between.""" """Read response stream until SQ_EOT, allowing common tags in between."""
reader = _SocketReader(self._conn._sock) reader = _make_socket_reader(self._conn._sock)
while True: while True:
tag = reader.read_short() tag = reader.read_short()
if tag == MessageType.SQ_EOT: if tag == MessageType.SQ_EOT:
@ -1662,7 +2078,6 @@ class Cursor:
(e.g. table or column name for "not found" errors). Empty for (e.g. table or column name for "not found" errors). Empty for
most syntax errors. most syntax errors.
""" """
from ._protocol import ProtocolError
sqlcode = reader.read_short() sqlcode = reader.read_short()
isamcode = reader.read_short() isamcode = reader.read_short()
@ -1680,7 +2095,7 @@ class Cursor:
if name_len & 1: if name_len & 1:
reader.read_exact(1) # pad to even reader.read_exact(1) # pad to even
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace") near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
except (ProtocolError, OSError): except WIRE_ERRORS:
pass pass
# Drain remaining bytes until SQ_EOT. Phase 28: a (ProtocolError, # Drain remaining bytes until SQ_EOT. Phase 28: a (ProtocolError,
# OSError) during drain means the wire is in an unknown state — # OSError) during drain means the wire is in an unknown state —
@ -1692,7 +2107,7 @@ class Cursor:
t = reader.read_short() t = reader.read_short()
if t == MessageType.SQ_EOT: if t == MessageType.SQ_EOT:
break break
except (ProtocolError, OSError): except WIRE_ERRORS:
with contextlib.suppress(Exception): with contextlib.suppress(Exception):
self._conn.close() self._conn.close()

0
src/informix_db/py.typed Normal file
View File

180
src/informix_db/rows.py Normal file
View File

@ -0,0 +1,180 @@
"""Rows that can be read by position, by column name, or by attribute.
PEP 249 only requires a sequence, and a sequence is what the driver
returns by default. That is fine for ``SELECT a, b`` and steadily worse
as the projection grows: ``row[11]`` tells a reader nothing, and stays
correct only until somebody adds a column in the middle.
Opting in with ``row_factory=Row`` gives the shape ``pyodbc`` and
``mssql-python`` provide, all three at once::
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur.execute("SELECT tabid, tabname FROM systables")
row = cur.fetchone()
row[0], row["tabname"], row.tabname
It is opt-in rather than the default because it is not free, and the
driver's whole argument is that pure Python can stay within noise of the
C driver on bulk fetch. Measured on a 20,000-row five-column fetch:
**37.2 ms with tuples, 40.8 ms with Row**, so about 9%. Defaulting it on
would move the published 1.05-1.15x ratio against IfxPy to roughly
1.15-1.25x, which is not a trade to make on everybody's behalf.
Where the 9% goes, against a plain tuple: about 30 ns per row to
construct, about 39 ns per ``row[0]`` because supporting ``row["name"]``
means ``__getitem__`` is a Python method rather than C-level tuple
indexing, and about 13 ns per unpack since CPython's fast path for
``a, b = row`` applies to exact tuples and not to subclasses.
Worth it for readability in application code. Not worth paying in a bulk
export that never looks at a column by name, which is exactly why it is
a choice.
``Row`` subclasses ``tuple``, so ``row == (1, "x")`` is still true and
existing code keeps working unchanged.
**A column always beats a method of the same name.** ``tuple`` defines
``count`` and ``index``; this class adds ``keys``, ``_asdict`` and
``_fields``. A column named any of those would otherwise resolve to the
method and hand back a bound method instead of a value, which is exactly
the shape of the framing bugs this driver spent a fortnight removing: a
plausible-looking wrong answer, in silence. So every such name gets a
descriptor and the column wins, and the reserved set is *computed* from
the class rather than hand-listed, because hand-listing it is what
missed ``keys``, ``_asdict`` and ``_fields`` the first time round. The
methods stay reachable through the class: ``tuple.count(row, x)``,
``Row.keys(row)``.
The machinery itself is name-mangled (``__fields`` / ``__map``) so that
shadowing ``_fields`` cannot break ``repr`` or ``_asdict``.
Column names come from ``cursor.description``. Informix folds unquoted
identifiers to lower case, so ``SELECT Config_Key`` is reachable as
``row.config_key``. Expressions get server-generated names that are not
Python identifiers, like ``(count(*))``; those are reachable by
subscript but not as attributes. Duplicate names resolve to the first
occurrence, matching ``pyodbc``. Dunder names cannot be shadowed and
stay subscript-only, which no real schema should notice.
"""
from __future__ import annotations
import operator
from functools import lru_cache
from typing import ClassVar
__all__ = ["Row", "make_row_class"]
class Row(tuple):
"""A result row addressable by position, name, or attribute.
Used as a ``row_factory``. The concrete class handed to each result
set is a subclass carrying that query's column names, built by
:func:`make_row_class`.
"""
__slots__ = ()
# Name-mangled to ``_Row__fields`` / ``_Row__map`` so that a column
# called "_fields" can be shadowed without breaking the machinery
# that reads it. Set on the per-result-set subclass.
__fields: ClassVar[tuple[str, ...]] = ()
__map: ClassVar[dict[str, int]] = {}
def __getitem__(self, key):
# ``key.__class__ is str`` rather than isinstance: this runs on
# every subscript.
if key.__class__ is str:
try:
return tuple.__getitem__(self, self.__map[key])
except KeyError:
raise KeyError(
f"no column named {key!r}; this row has "
f"{list(self.__fields)}"
) from None
return tuple.__getitem__(self, key)
def __getattr__(self, name):
# Only reached when normal attribute lookup has already failed,
# so this costs nothing for names that do not collide.
try:
return tuple.__getitem__(self, self.__map[name])
except KeyError:
raise AttributeError(
f"no column named {name!r}; this row has "
f"{list(self.__fields)}"
) from None
@property
def _fields(self) -> tuple[str, ...]:
"""Column names, in select order. Mirrors ``namedtuple._fields``."""
return self.__fields
def keys(self) -> tuple[str, ...]:
"""Column names, in select order."""
return self.__fields
def _asdict(self) -> dict:
"""A plain ``dict`` of the row.
On duplicate column names the last occurrence wins here, while
subscript access gives the first. A dict cannot represent both,
and quietly dropping a duplicate is better than raising on a
query that is otherwise fine.
"""
return dict(zip(self.__fields, self, strict=True))
def __repr__(self) -> str:
fields = self.__fields
if len(fields) != len(self):
return tuple.__repr__(self)
body = ", ".join(
f"{name}={value!r}"
for name, value in zip(fields, self, strict=True)
)
return f"Row({body})"
def __reduce__(self):
# The per-result-set class is created at runtime and cannot be
# pickled by reference, so rebuild it from the field names.
return (_rebuild_row, (self.__fields, tuple(self)))
# Every non-dunder attribute a Row already answers to. A column with one
# of these names gets a descriptor so the column wins. Computed, not
# hand-listed: the hand-listed version covered ``count`` and ``index``
# and silently missed ``keys``, ``_asdict`` and ``_fields``.
_RESERVED = frozenset(
name for name in dir(Row) if not name.startswith("__")
) - {"_Row__fields", "_Row__map"}
def _rebuild_row(fields: tuple[str, ...], values: tuple):
return make_row_class(fields)(values)
@lru_cache(maxsize=256)
def make_row_class(fields: tuple[str, ...]) -> type[Row]:
"""Build (and cache) the row class for one column-name shape.
Cached because a class per ``execute()`` would put a ``type()`` call
on the path of every small query, and applications run the same
handful of statement shapes over and over. Keyed on the names alone,
so two queries selecting the same columns share a class.
"""
namespace: dict = {
"__slots__": (),
"_Row__fields": fields,
# First occurrence wins on duplicates, matching pyodbc. Building
# the map in reverse and letting earlier entries overwrite later
# ones is the shortest way to say that.
"_Row__map": {
name: i for i, name in reversed(list(enumerate(fields)))
},
}
for i, name in enumerate(fields):
if name in _RESERVED:
# A column of this name would otherwise resolve to a method.
namespace[name] = property(operator.itemgetter(i))
return type("Row", (Row,), namespace)

View File

@ -1,6 +1,6 @@
{ {
"machine_info": { "machine_info": {
"node": "rpm-bullet", "node": "PLACEHOLDER",
"processor": "", "processor": "",
"machine": "x86_64", "machine": "x86_64",
"python_compiler": "Clang 22.1.1 ", "python_compiler": "Clang 22.1.1 ",

View File

@ -0,0 +1,48 @@
# Older Informix versions, for the server-compatibility matrix in the README.
#
# The primary dev container (tests/docker-compose.yml) runs Informix 15 on
# 9088. These two run alongside it on 9089 / 9090 so all three can be tested
# without tearing anything down.
#
# These images live on Docker Hub under ibmcom/ rather than icr.io/informix/ —
# IBM stopped publishing the older developer editions to the newer registry.
# They're tagged rather than digest-pinned because the ibmcom repo is frozen
# (no new pushes since the icr.io move), so the tags are already immutable in
# practice.
#
# make ifx-legacy-up start both
# make ifx-legacy-setup create blobspace1 + sbspace1 (needed for LOB tests)
# make test-matrix run the integration suite against all three
# make ifx-legacy-down stop and remove
services:
ifx1210:
container_name: informix-db-test-1210
image: ibmcom/informix-developer-database:12.10.FC12W1DE
privileged: true
environment:
LICENSE: accept
SIZE: small
ports:
- "9089:9088"
healthcheck:
test: ["CMD-SHELL", "exec 3<>/dev/tcp/127.0.0.1/9088 && exec 3<&- && exec 3>&-"]
interval: 5s
timeout: 3s
retries: 60
start_period: 30s
ifx1410:
container_name: informix-db-test-1410
image: ibmcom/informix-developer-database:14.10.FC7W1DE
privileged: true
environment:
LICENSE: accept
SIZE: small
ports:
- "9090:9088"
healthcheck:
test: ["CMD-SHELL", "exec 3<>/dev/tcp/127.0.0.1/9088 && exec 3<&- && exec 3>&-"]
interval: 5s
timeout: 3s
retries: 60
start_period: 30s

76
tests/setup-spaces.sh Executable file
View File

@ -0,0 +1,76 @@
#!/usr/bin/env bash
# Create blobspace1 + sbspace1 in an Informix dev container.
#
# The developer-edition images ship with only rootdbs. BYTE/TEXT tests need a
# blobspace; BLOB/CLOB (smart-LOB) tests need an sbspace plus SBSPACENAME set.
# Without them, ~21 integration tests fail with errors that look like driver
# bugs but are pure server configuration.
#
# Usage: tests/setup-spaces.sh <container-name>
#
# Idempotent: re-running against a container that already has the spaces prints
# the server's "already exists" complaint and exits 0.
#
# Version quirks this handles, learned the hard way:
# * INFORMIXDIR differs — 12.10/14.10 use /opt/ibm/informix, 15 nests a
# versioned subdirectory (/opt/ibm/informix/v15.0.1.0.3).
# * ONCONFIG is named `onconfig` on some images and `onconfig.informix` on
# others; we probe for the file rather than guessing.
# * `bash -lc` wipes INFORMIXDIR from the environment on these images, which
# makes every utility fail with "Unable to read $INFORMIXDIR (/usr/informix)".
# Use `bash -c` and export explicitly.
# * 12.10 DE ships no `ontape`. The level-0 archive turns out to be
# unnecessary for the tests, so we attempt it and shrug if it's missing.
set -euo pipefail
CONTAINER="${1:?usage: setup-spaces.sh <container-name>}"
docker exec -u informix "$CONTAINER" bash -c '
set -u
# Locate INFORMIXDIR: either /opt/ibm/informix or a versioned subdir under it.
for cand in /opt/ibm/informix /opt/ibm/informix/v*; do
if [ -x "$cand/bin/onspaces" ]; then
export INFORMIXDIR="$cand"
break
fi
done
if [ -z "${INFORMIXDIR:-}" ]; then
echo "could not locate INFORMIXDIR (no bin/onspaces found)" >&2
exit 1
fi
export PATH="$INFORMIXDIR/bin:$PATH"
export INFORMIXSERVER=informix
export INFORMIXSQLHOSTS="$INFORMIXDIR/etc/sqlhosts"
# ONCONFIG name varies across images.
for cfg in onconfig onconfig.informix; do
if [ -f "$INFORMIXDIR/etc/$cfg" ]; then
export ONCONFIG="$cfg"
break
fi
done
: "${ONCONFIG:?no onconfig found in $INFORMIXDIR/etc}"
echo "INFORMIXDIR=$INFORMIXDIR ONCONFIG=$ONCONFIG"
SPACES=/opt/ibm/data/spaces
mkdir -p "$SPACES"
for f in blobspace1 sbspace1; do
[ -e "$SPACES/$f" ] || : > "$SPACES/$f"
chmod 660 "$SPACES/$f"
done
onspaces -c -b blobspace1 -g 1 -p "$SPACES/blobspace1" -o 0 -s 50000 2>&1 | grep -vE "^\s*$" || true
onspaces -c -S sbspace1 -p "$SPACES/sbspace1" -o 0 -s 50000 -Df "AVG_LO_SIZE=100" 2>&1 | grep -vE "^\s*$" || true
onmode -wm SBSPACENAME=sbspace1 2>&1 | tail -1 || true
# Level-0 archive. Informix warns it is required after adding a space; in
# practice the tests pass without it, and 12.10 DE has no ontape at all.
onmode -wm TAPEDEV=/dev/null >/dev/null 2>&1 || true
if command -v ontape >/dev/null 2>&1; then
ontape -s -L 0 2>&1 | tail -1 || true
else
echo "ontape not present on this image; skipping level-0 archive"
fi
'

265
tests/test_async_threads.py Normal file
View File

@ -0,0 +1,265 @@
"""The async layer used a thread pool it shared with the whole process.
``asyncio.to_thread`` runs on the event loop's default executor. That
executor belongs to the process, not to us, and it is sized from the CPU
count ``min(32, cpu_count + 4)``, which is six threads on a two-CPU
container. Every blocking call in ``informix_db.aio`` went through it.
Two consequences, both silent.
**Cancelled calls hold their threads.** ``asyncio.to_thread`` cannot
interrupt a worker, so a cancelled await leaves the thread running the
wire call until the read timeout expires. Cancellation is ordinary in a
web app a client disconnect cancels the request task so a handful of
them pins every thread in the shared pool. Unrelated ``to_thread`` work
anywhere else in the process then stops dead, and so does the driver.
**Pool concurrency was capped by an unrelated number.** A pool with
``max_size=20`` on a two-CPU box ran six queries at a time, and nothing
said so.
Each connection now owns one thread. That is the right size rather than
a compromise: the sync connection serializes every wire operation on its
own lock, so a second thread could do nothing but wait for the first. It
also avoids a deadlock that a shared pool-sized executor invites with
N threads and N connections, N tasks blocked in ``acquire`` occupy every
thread while the connection they wait for is held by a task that now
needs a thread to finish and release it.
"""
from __future__ import annotations
import asyncio
import threading
import time
from concurrent.futures import ThreadPoolExecutor
import pytest
from informix_db import aio
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _kw(conn_params: ConnParams) -> dict:
return {
"host": conn_params.host,
"port": conn_params.port,
"user": conn_params.user,
"password": conn_params.password,
"database": conn_params.database,
"server": conn_params.server,
"connect_timeout": 10.0,
"read_timeout": 20.0,
"autocommit": True,
}
@pytest.mark.asyncio
async def test_queries_run_with_the_default_executor_fully_held(
conn_params: ConnParams,
) -> None:
"""The starvation case, with real queries. Four unrelated jobs occupy
every thread of a four-worker default executor; the driver must not
be waiting behind them."""
loop = asyncio.get_running_loop()
previous = loop._default_executor
loop.set_default_executor(ThreadPoolExecutor(max_workers=4))
conns = [await aio.connect(**_kw(conn_params)) for _ in range(4)]
cursors = [await c.cursor() for c in conns]
gate = threading.Event()
hogs = [
asyncio.create_task(asyncio.to_thread(gate.wait, 30)) for _ in range(4)
]
await asyncio.sleep(0.3)
async def query(cursor, i: int):
await cursor.execute(f"SELECT FIRST 1 tabid + {i} FROM systables")
return await cursor.fetchone()
try:
rows = await asyncio.wait_for(
asyncio.gather(*(query(c, i) for i, c in enumerate(cursors))),
timeout=10.0,
)
assert len(rows) == 4
assert all(r is not None for r in rows)
finally:
gate.set()
await asyncio.gather(*hogs, return_exceptions=True)
for c in conns:
await c.close()
# set_default_executor rejects None, which is what the loop
# starts with before anything has used to_thread.
loop.set_default_executor(previous or ThreadPoolExecutor())
@pytest.mark.asyncio
async def test_each_connection_gets_its_own_thread(
conn_params: ConnParams,
) -> None:
a = await aio.connect(**_kw(conn_params))
b = await aio.connect(**_kw(conn_params))
try:
assert a._executor is not b._executor
assert a._executor._max_workers == 1, (
"more than one thread per connection cannot help — the wire "
"lock serializes them anyway"
)
finally:
await a.close()
await b.close()
@pytest.mark.asyncio
async def test_cursor_runs_on_its_connections_thread(
conn_params: ConnParams,
) -> None:
"""A cursor must not fall back to the default executor, or half the
work goes back to being shared."""
conn = await aio.connect(**_kw(conn_params))
try:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
names: list[str] = []
await conn._run(lambda: names.append(threading.current_thread().name))
assert names[0].startswith("informix-conn")
assert cur._run.__self__ is conn
finally:
await conn.close()
@pytest.mark.asyncio
async def test_pool_reuses_one_thread_per_connection(
conn_params: ConnParams,
) -> None:
"""The executor lives on the sync connection, so a connection handed
out, returned, and handed out again keeps the same thread instead of
spawning one per acquire."""
pool = await aio.create_pool(**_kw(conn_params), min_size=1, max_size=2)
try:
seen = []
for _ in range(4):
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
assert await cur.fetchone() is not None
seen.append(id(conn._executor))
assert len(set(seen)) <= 2, (
f"expected at most one executor per pooled connection, saw "
f"{len(set(seen))}"
)
finally:
await pool.close()
@pytest.mark.asyncio
async def test_pool_concurrency_is_not_capped_by_the_default_executor(
conn_params: ConnParams,
) -> None:
"""Six concurrent pooled queries against a two-worker default
executor. Under the old arrangement at most two could run."""
loop = asyncio.get_running_loop()
previous = loop._default_executor
pool = await aio.create_pool(**_kw(conn_params), min_size=6, max_size=6)
loop.set_default_executor(ThreadPoolExecutor(max_workers=2))
try:
async def one(i: int):
async with pool.connection(timeout=15.0) as conn:
cur = await conn.cursor()
# Long enough that serialized execution would be obvious.
await cur.execute(
"SELECT FIRST 200 a.tabid FROM systables a, systables b"
)
return len(await cur.fetchall())
started = time.monotonic()
results = await asyncio.wait_for(
asyncio.gather(*(one(i) for i in range(6))), timeout=25.0
)
assert all(r > 0 for r in results)
assert len(results) == 6
elapsed = time.monotonic() - started
assert elapsed < 20.0, f"queries appear serialized ({elapsed:.1f}s)"
finally:
await pool.close()
# set_default_executor rejects None, which is what the loop
# starts with before anything has used to_thread.
loop.set_default_executor(previous or ThreadPoolExecutor())
@pytest.mark.asyncio
async def test_closing_a_connection_stops_its_thread(
conn_params: ConnParams,
) -> None:
"""One thread per connection is only affordable if the thread goes
away with the connection."""
before = threading.active_count()
conns = [await aio.connect(**_kw(conn_params)) for _ in range(5)]
for c in conns:
cur = await c.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
for c in conns:
await c.close()
del conns
# Threads exit asynchronously after shutdown; give them a moment.
for _ in range(50):
if threading.active_count() <= before + 1:
break
await asyncio.sleep(0.1)
assert threading.active_count() <= before + 1, (
f"connection threads outlived their connections "
f"({before} -> {threading.active_count()})"
)
@pytest.mark.asyncio
async def test_row_factory_reaches_the_async_paths(
conn_params: ConnParams,
) -> None:
"""The async layer wraps the sync cursor, so row_factory should flow
through untouched. Each of the four async fetch routes is a separate
call site, and async iteration goes through __anext__ rather than
__next__."""
import informix_db
conn = await aio.connect(row_factory=informix_db.Row, **_kw(conn_params))
try:
cur = await conn.cursor()
sql = "SELECT FIRST 2 tabid, tabname FROM systables ORDER BY tabid"
await cur.execute(sql)
one = await cur.fetchone()
assert one[0] == one["tabid"] == one.tabid
await cur.execute(sql)
assert [r.tabid for r in await cur.fetchall()] == [1, 2]
await cur.execute(sql)
assert all(r.tabname for r in await cur.fetchmany(2))
await cur.execute(sql)
assert [r.tabid async for r in cur] == [1, 2]
finally:
await conn.close()
@pytest.mark.asyncio
async def test_async_pool_forwards_the_row_factory(
conn_params: ConnParams,
) -> None:
import informix_db
pool = await aio.create_pool(
row_factory=informix_db.Row, min_size=1, max_size=2, **_kw(conn_params)
)
try:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
assert (await cur.fetchone()).tabid == 1
finally:
await pool.close()

View File

@ -0,0 +1,258 @@
"""Regression tests for pipelined executemany, scrollable cursors, and LOBs.
Found here by fuzzing: **a client-side encoding failure inside an
``executemany`` batch leaked the prepared statement.** The PDUs are built
after the PREPARE, so a value the connection's codec cannot represent
raises there before anything is drained and the exception escaped
without the RELEASE. The leaked statement then collided with the next
PREPARE and every later call on that connection failed with an error
pointing at the *previous* SQL.
``_execute_dml_with_params`` already guarded exactly this case for the
single-row path. The pipelined path was simply missed, which is the
recurring shape of these bugs: a hazard understood in one place and not
carried to its sibling.
The rest of this file is the coverage that proved the neighbouring paths
sound batch failures at every position, scroll boundaries, LOB sizes
kept so they stay that way. Each failure case ends in a health check,
because the interesting damage is never in the statement that failed.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=15.0,
read_timeout=30.0,
**kw,
)
def _assert_healthy(cur) -> None:
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, "connection unusable"
# ---------------------------------------------------------------------------
# executemany — the encoding-failure leak, and the neighbours
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [3, 50])
@pytest.mark.parametrize("position", ["first", "mid", "last"])
def test_encoding_failure_in_batch_does_not_leak(
conn_params: ConnParams, n: int, position: str
) -> None:
"""The bug. A value the codec can't encode raises after the PREPARE;
escaping without the RELEASE bricked the connection."""
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_enc (s VARCHAR(32))")
rows: list[tuple] = [(f"s{i}",) for i in range(n)]
rows[idx] = ("中文",) # not representable in iso-8859-1
with pytest.raises(Exception): # noqa: B017 — DataError or UnicodeError
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
# The connection must survive, repeatedly.
for _ in range(3):
_assert_healthy(cur)
with pytest.raises(Exception): # noqa: B017
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
_assert_healthy(cur)
@pytest.mark.parametrize("n", [2, 3, 10, 100])
@pytest.mark.parametrize("position", ["first", "mid", "last"])
def test_constraint_violation_in_batch_recovers(
conn_params: ConnParams, n: int, position: str
) -> None:
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_dup (k INT PRIMARY KEY)")
cur.execute("INSERT INTO t_em_dup VALUES (?)", (10_000,))
rows = [(i,) for i in range(n)]
rows[idx] = (10_000,)
with pytest.raises(informix_db.Error):
cur.executemany("INSERT INTO t_em_dup VALUES (?)", rows)
_assert_healthy(cur)
# Whatever the partial-batch semantics, COUNT(*) and a full fetch
# must agree — a disagreement means the row decoder and the server
# have different ideas about what is in the table.
cur.execute("SELECT COUNT(*) FROM t_em_dup")
(counted,) = cur.fetchone()
cur.execute("SELECT k FROM t_em_dup")
assert counted == len(cur.fetchall())
@pytest.mark.parametrize("n", [1, 2, 3, 10, 100, 1000])
def test_executemany_inserts_exactly_n_rows(
conn_params: ConnParams, n: int
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_ok (k INT, v VARCHAR(16))")
cur.executemany(
"INSERT INTO t_em_ok VALUES (?, ?)",
[(i, f"v{i}") for i in range(n)],
)
cur.execute("SELECT COUNT(*) FROM t_em_ok")
assert cur.fetchone() == (n,)
def test_executemany_empty_and_single(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_edge (k INT)")
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [])
_assert_healthy(cur)
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [(1,)])
cur.execute("SELECT COUNT(*) FROM t_em_edge")
assert cur.fetchone() == (1,)
# ---------------------------------------------------------------------------
# Scrollable cursors — boundaries get their own round-trip each
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [0, 1, 2, 5, 50, 300])
def test_scroll_boundaries(conn_params: ConnParams, n: int) -> None:
with _connect(conn_params, autocommit=True) as conn:
setup = conn.cursor()
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll")
setup.execute("CREATE TABLE t_scroll (k INT)")
try:
if n:
setup.executemany(
"INSERT INTO t_scroll VALUES (?)", [(i,) for i in range(n)]
)
cur = conn.cursor(scrollable=True)
try:
cur.execute("SELECT k FROM t_scroll ORDER BY k")
assert cur.fetch_first() == ((0,) if n else None)
assert cur.fetch_last() == ((n - 1,) if n else None)
if n >= 3:
assert cur.fetch_absolute(2) == (2,)
assert cur.fetch_prior() == (1,)
assert cur.fetch_relative(2) == (3,)
# Off both ends must be None — not a crash, not a wrap.
assert cur.fetch_absolute(n + 50) is None
cur.fetch_first()
assert cur.fetch_prior() is None
if n:
cur.fetch_first()
seen = [(0,)]
while (row := cur.fetchone()) is not None:
seen.append(row)
assert seen == [(i,) for i in range(n)]
finally:
cur.close()
_assert_healthy(setup)
finally:
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll")
def test_abandoned_scroll_cursors_do_not_leak(
conn_params: ConnParams,
) -> None:
"""Scroll cursors stay open server-side, so abandoning one leaks
unless the finalizer runs."""
with _connect(conn_params, autocommit=True) as conn:
setup = conn.cursor()
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll_ab")
setup.execute("CREATE TABLE t_scroll_ab (k INT)")
try:
setup.executemany(
"INSERT INTO t_scroll_ab VALUES (?)", [(i,) for i in range(20)]
)
for i in range(20):
c = conn.cursor(scrollable=True)
c.execute("SELECT k FROM t_scroll_ab ORDER BY k")
c.fetch_first()
if i % 2:
c.close()
else:
del c # rely on the finalizer
_assert_healthy(setup)
setup.execute("SELECT COUNT(*) FROM t_scroll_ab")
assert setup.fetchone() == (20,)
finally:
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll_ab")
# ---------------------------------------------------------------------------
# Smart LOBs
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
"size", [0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535, 65536]
)
def test_blob_round_trip_sizes(conn_params: ConnParams, size: int) -> None:
"""Sizes straddle the 4096-byte SQ_FILE chunk and the 64K mark."""
payload = bytes((i * 7 + size) % 256 for i in range(size))
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_blob_rt")
try:
cur.execute("CREATE TABLE t_blob_rt (k INT, b BLOB)")
except informix_db.Error:
pytest.skip("no sbspace configured; see make ifx-spaces")
try:
cur.write_blob_column(
"INSERT INTO t_blob_rt VALUES (?, BLOB_PLACEHOLDER)",
payload, (1,),
)
got = cur.read_blob_column(
"SELECT b FROM t_blob_rt WHERE k = ?", (1,)
)
if size == 0:
assert got in (b"", None)
else:
assert got == payload
_assert_healthy(cur)
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_blob_rt")
def test_failed_blob_read_recovers(conn_params: ConnParams) -> None:
"""The SQ_FILE path involves a server-side temp file; a failure part
way through must not strand the connection."""
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
for _ in range(5):
with contextlib.suppress(Exception):
cur.read_blob_column("SELECT b FROM t_no_such_blob_tbl", ())
_assert_healthy(cur)

144
tests/test_capabilities.py Normal file
View File

@ -0,0 +1,144 @@
"""Integration tests for live SQ_PROTOCOLS negotiation.
These run against whatever server ``IFX_PORT`` points at, so `make
test-matrix` exercises them on 12.10, 14.10, and 15 in turn.
The important one is ``test_no_violated_assumptions``: this driver
hardcodes several wire-framing choices that SQLI actually negotiates, and
a mismatch corrupts rows silently. That test turns "we assume this" into
"we check this on every supported server".
"""
from __future__ import annotations
import pytest
import informix_db
from informix_db._capabilities import ENHANCED_PROTOCOL_CAP, ServerCapabilities
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
def test_capabilities_are_decoded(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
caps = conn.server_capabilities
assert isinstance(caps, ServerCapabilities)
assert caps.raw_mask, "server sent an empty protocols reply"
assert caps.bits
def test_no_violated_assumptions(conn_params: ConnParams) -> None:
"""Every wire-framing shape we hardcode is one the server negotiated.
If this fails, rows are being decoded against the wrong framing and
the fix is to branch on the capability rather than assume it.
"""
with _connect(conn_params) as conn:
caps = conn.server_capabilities
assert caps is not None
assert caps.violated_assumptions() == []
def test_enhanced_protocol_negotiated(conn_params: ConnParams) -> None:
"""Cap_1 is the client's declared protocol level echoed back. Every
supported server accepts 316."""
with _connect(conn_params) as conn:
caps = conn.server_capabilities
assert caps is not None
assert caps.cap_1 == ENHANCED_PROTOCOL_CAP
assert caps.enhanced_protocol
def test_framing_capabilities_present(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
caps = conn.server_capabilities
assert caps is not None
assert caps.four_byte_offset
assert caps.varchar_var_len
assert caps.remove_64k_limit
assert caps.usver
def test_server_version_is_exposed(conn_params: ConnParams) -> None:
"""Assert the shape rather than a value so this holds across the matrix."""
with _connect(conn_params) as conn:
version = conn.server_version
assert "Informix" in version
assert "Version" in version
def test_server_version_reports_release_not_protocol_version(
conn_params: ConnParams,
) -> None:
"""A field report flagged ``server_version`` as looking like a
client-SDK version: the login response announces 12.10 servers as 9.56
and 14.10 as 9.59. ``server_version`` now answers with the release; the
raw login string moved to ``server_version_internal``."""
with _connect(conn_params) as conn:
release = conn.server_version
internal = conn.server_version_internal
assert "Informix" in internal
# On 12.10/14.10 these genuinely differ; on 15 they agree.
if "9.5" in internal:
assert release != internal
assert "9.5" not in release, (
f"server_version still reports the protocol version: {release!r}"
)
def test_server_version_is_cached(conn_params: ConnParams) -> None:
"""It costs a round-trip, so repeated access must not repeat it."""
with _connect(conn_params) as conn:
assert conn.server_version == conn.server_version
assert conn._server_version_full is not None
def test_server_version_degrades_without_a_database(
conn_params: ConnParams,
) -> None:
"""The DBINFO lookup needs an open database. With none, the property
must fall back rather than raise a version lookup should never be
able to break a working connection."""
conn = informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=None,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
try:
assert isinstance(conn.server_version, str)
finally:
conn.close()
def test_capabilities_survive_multiple_connections(
conn_params: ConnParams,
) -> None:
"""Negotiation happens per-connection; two connections to the same
server must agree."""
with _connect(conn_params) as first, _connect(conn_params) as second:
assert first.server_capabilities is not None
assert second.server_capabilities is not None
assert (
first.server_capabilities.raw_mask
== second.server_capabilities.raw_mask
)
assert first.server_capabilities.bits == second.server_capabilities.bits

View File

@ -0,0 +1,200 @@
"""Unit tests for SQ_PROTOCOLS capability decoding — no server required.
The masks below were captured from live servers on 2026-08-27. Their most
interesting property is that the first eight bytes are identical across
Informix 12.10, 14.10, and 15 which is why those three releases speak an
indistinguishable SQLI dialect.
"""
from __future__ import annotations
import dataclasses
import pytest
from informix_db._capabilities import (
BIT_FOUR_BYTE_OFFSET,
BIT_REMOVE_64K_LIMIT,
BIT_USVER,
BIT_VARCHAR_VAR_LEN,
CLIENT_PROTOCOLS_MASK,
ENHANCED_PROTOCOL_CAP,
ServerCapabilities,
_decode_bits,
)
MASK_15 = bytes.fromhex("bdbe9ffe7fb7ffefff")
MASK_1410 = bytes.fromhex("bdbe9ffe7fb7ffeff8")
MASK_1210 = bytes.fromhex("bdbe9ffe7fb7ffeff0")
ALL_MASKS = [
pytest.param(MASK_15, id="15.0.1.0.3"),
pytest.param(MASK_1410, id="14.10.FC7W1"),
pytest.param(MASK_1210, id="12.10.FC12W1DE"),
]
# --------------------------------------------------------------------------
# Bit expansion
# --------------------------------------------------------------------------
def test_bit_numbering_is_msb_first() -> None:
assert _decode_bits(b"\x80") == {0}
assert _decode_bits(b"\x01") == {7}
assert _decode_bits(b"\x00\x80") == {8}
assert _decode_bits(b"\x00\x01") == {15}
def test_bit_expansion_of_first_captured_byte() -> None:
"""0xBD is 1011 1101 -> bits 0, 2, 3, 4, 5, 7."""
assert _decode_bits(b"\xbd") == {0, 2, 3, 4, 5, 7}
def test_empty_mask_yields_no_bits() -> None:
assert _decode_bits(b"") == set()
# --------------------------------------------------------------------------
# The captured masks
# --------------------------------------------------------------------------
def test_first_eight_bytes_identical_across_versions() -> None:
"""The finding this whole module exists to record: 12.10, 14.10, and 15
negotiate exactly the same 64-bit capability set. Only the 9th byte
which IBM's own JDBC driver discards — differs."""
assert MASK_15[:8] == MASK_1410[:8] == MASK_1210[:8]
assert len({MASK_15[8], MASK_1410[8], MASK_1210[8]}) == 3
@pytest.mark.parametrize("mask", ALL_MASKS)
def test_framing_bits_set_on_every_tested_server(mask: bytes) -> None:
"""Every wire-framing choice this driver hardcodes is one the server
actually negotiated. If this ever fails, the row decoder is wrong."""
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
assert caps.four_byte_offset, "describe offsets would be misparsed"
assert caps.varchar_var_len, "VARCHAR framing would be misparsed"
assert caps.remove_64k_limit, "fast-path length prefix would be wrong"
assert caps.bigint
assert caps.long_id
assert caps.lvarchar_gt_2k
assert caps.gls
@pytest.mark.parametrize("mask", ALL_MASKS)
def test_no_violated_assumptions_on_tested_servers(mask: bytes) -> None:
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
assert caps.violated_assumptions() == []
@pytest.mark.parametrize("mask", ALL_MASKS)
def test_ninth_byte_is_decoded(mask: bytes) -> None:
"""JDBC drops bits 64+; we keep them because they're the only part
that varies between releases."""
caps = ServerCapabilities.from_wire(mask, cap_1=ENHANCED_PROTOCOL_CAP)
assert any(b >= 64 for b in caps.bits)
# --------------------------------------------------------------------------
# Pre-set bits — the reason isUSVER is a dead end
# --------------------------------------------------------------------------
def test_preset_bits_applied_when_cap1_nonzero() -> None:
"""JDBC pre-sets {0,2,3,4,49,51} for any non-zero Cap_1 and BitSet.set
never clears, so these are true on any modern server regardless of the
mask. That's why isUSVER never varies and can't explain version-specific
behaviour."""
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=ENHANCED_PROTOCOL_CAP)
assert {0, 2, 3, 4, 49, 51}.issubset(caps.bits)
assert caps.usver
def test_preset_bits_not_applied_when_cap1_zero() -> None:
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=0)
assert caps.bits == frozenset()
assert not caps.usver
def test_preset_cannot_clear_a_bit_the_mask_set() -> None:
caps = ServerCapabilities.from_wire(MASK_15, cap_1=ENHANCED_PROTOCOL_CAP)
assert caps.has(BIT_USVER)
assert caps.has(BIT_FOUR_BYTE_OFFSET)
# --------------------------------------------------------------------------
# Assumption violations — the diagnostic path
# --------------------------------------------------------------------------
def test_missing_four_byte_offset_is_reported() -> None:
# Everything set except bit 50.
bits = bytearray(b"\xff" * 9)
bits[BIT_FOUR_BYTE_OFFSET // 8] &= ~(0x80 >> (BIT_FOUR_BYTE_OFFSET % 8)) & 0xFF
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
problems = caps.violated_assumptions()
assert len(problems) == 1
assert "4-byte describe offsets" in problems[0]
def test_missing_varchar_var_len_is_reported() -> None:
bits = bytearray(b"\xff" * 9)
bits[BIT_VARCHAR_VAR_LEN // 8] &= ~(0x80 >> (BIT_VARCHAR_VAR_LEN % 8)) & 0xFF
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
problems = caps.violated_assumptions()
assert len(problems) == 1
assert "variable-length VARCHAR" in problems[0]
def test_missing_remove_64k_is_reported() -> None:
bits = bytearray(b"\xff" * 9)
bits[BIT_REMOVE_64K_LIMIT // 8] &= ~(0x80 >> (BIT_REMOVE_64K_LIMIT % 8)) & 0xFF
caps = ServerCapabilities.from_wire(bytes(bits), cap_1=ENHANCED_PROTOCOL_CAP)
problems = caps.violated_assumptions()
assert len(problems) == 1
assert "64K limit" in problems[0]
def test_all_zero_mask_reports_every_assumption() -> None:
caps = ServerCapabilities.from_wire(b"\x00" * 9, cap_1=ENHANCED_PROTOCOL_CAP)
assert len(caps.violated_assumptions()) == 3
# --------------------------------------------------------------------------
# Misc
# --------------------------------------------------------------------------
def test_client_offer_matches_jdbc_reference() -> None:
"""IfxSqliConnect.clientProtocols, verbatim. Changing this changes what
the server negotiates, so it is pinned."""
jdbc_reference = bytes([0xFF, 0xFC, 0x7F, 0xFC, 0x3C, 0x8C, 0xAA, 0x97])
assert jdbc_reference == CLIENT_PROTOCOLS_MASK
assert len(CLIENT_PROTOCOLS_MASK) == 8
def test_enhanced_protocol_requires_exact_cap1() -> None:
"""JDBC tests == 316, not >=, because the server echoes the client's own
declared level rather than reporting its own version."""
assert ServerCapabilities.from_wire(MASK_15, cap_1=316).enhanced_protocol
assert not ServerCapabilities.from_wire(MASK_15, cap_1=315).enhanced_protocol
assert not ServerCapabilities.from_wire(MASK_15, cap_1=317).enhanced_protocol
def test_repr_is_readable_and_includes_mask() -> None:
caps = ServerCapabilities.from_wire(
MASK_15, cap_1=316, server_version="IBM Informix Dynamic Server Version 15"
)
text = repr(caps)
assert "bdbe9ffe7fb7ffefff" in text
assert "usver" in text
assert "cap_1=316" in text
def test_capabilities_are_frozen() -> None:
"""Immutable so a connection's negotiated state can't be edited after
the fact and quietly disagree with what's on the wire."""
caps = ServerCapabilities.from_wire(MASK_15, cap_1=316)
with pytest.raises(dataclasses.FrozenInstanceError):
caps.cap_1 = 1 # type: ignore[misc]

View File

@ -0,0 +1,151 @@
"""Regression tests for DATETIME sub-second precision on bind.
``_encode_datetime`` used to emit YEAR TO SECOND unconditionally, so
binding a ``datetime`` carrying microseconds into a
``DATETIME YEAR TO FRACTION(n)`` column stored zeros silently, with no
error and no warning. Reads were always fine, which is what made it hard
to notice: the value only went missing on the way in.
It now emits YEAR TO FRACTION(5) when ``microsecond`` is non-zero and
keeps the original YEAR TO SECOND encoding otherwise.
FRACTION(5) is Informix's widest and resolves to 10 µs, so Python's
sixth microsecond digit is dropped. That's a real limit of the type, not
a driver choice, and the tests below pin the truncation so it can't drift
into something worse.
"""
from __future__ import annotations
import datetime
import pytest
import informix_db
from informix_db.converters import _encode_datetime
from tests.conftest import ConnParams
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
# --------------------------------------------------------------------------
# Encoder unit tests — no server needed
# --------------------------------------------------------------------------
def test_encoder_uses_year_to_second_without_microseconds() -> None:
"""The long-exercised path must stay byte-identical."""
type_code, prec, raw = _encode_datetime(
datetime.datetime(2026, 8, 31, 12, 30, 15)
)
assert type_code == 10
assert prec == (14 << 8) | 10 # digit_count 14, YEAR..SECOND
assert raw == b"\x00\x08\xc7\x14\x1a\x08\x1f\x0c\x1e\x0f"
assert len(raw) == 10 # 2 len + 1 exp + 7 BCD pairs
def test_encoder_widens_to_fraction_when_microseconds_present() -> None:
type_code, prec, raw = _encode_datetime(
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
)
assert type_code == 10
assert prec == (19 << 8) | 15 # digit_count 19, YEAR..FRACTION(5)
assert len(raw) == 13 # 2 len + 1 exp + 10 BCD pairs
# Exponent byte is unchanged: the integer part is still 7 base-100
# pairs, the fraction just adds three more after the point.
assert raw[2] == 0xC7
# Trailing pairs carry 120000 as BCD 12/00/00.
assert raw[-3:] == b"\x0c\x00\x00"
def test_encoder_pads_fraction_to_six_digits() -> None:
"""One microsecond must not shift the BCD pairs."""
_, prec, raw = _encode_datetime(
datetime.datetime(2026, 8, 31, 12, 30, 15, 1)
)
assert prec == (19 << 8) | 15
assert len(raw) == 13
assert raw[-3:] == b"\x00\x00\x01" # 000001
# --------------------------------------------------------------------------
# Round-trip against a real server
# --------------------------------------------------------------------------
@pytest.mark.integration
@pytest.mark.parametrize(
("microsecond", "expected"),
[
(0, 0),
(120000, 120000),
(500000, 500000),
(1, 0), # below FRACTION(5) resolution
(999999, 999990), # truncated to 5 significant digits
],
)
def test_fraction_round_trip(
conn_params: ConnParams, microsecond: int, expected: int
) -> None:
value = datetime.datetime(2026, 8, 31, 12, 30, 15, microsecond)
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_dt_frac "
"(k INT, t DATETIME YEAR TO FRACTION(5))"
)
cur.execute("INSERT INTO t_dt_frac VALUES (?, ?)", (1, value))
cur.execute("SELECT t FROM t_dt_frac")
(got,) = cur.fetchone()
assert got == value.replace(microsecond=expected)
@pytest.mark.integration
def test_fraction_bind_into_year_to_second_column(
conn_params: ConnParams,
) -> None:
"""Widening the bind must not break narrower columns — Informix
converts between qualifiers on assignment, truncating server-side."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_dt_sec (k INT, t DATETIME YEAR TO SECOND)"
)
cur.execute(
"INSERT INTO t_dt_sec VALUES (?, ?)",
(1, datetime.datetime(2026, 8, 31, 12, 30, 15, 987654)),
)
cur.execute("SELECT t FROM t_dt_sec")
assert cur.fetchone() == (
datetime.datetime(2026, 8, 31, 12, 30, 15),
)
@pytest.mark.integration
def test_fraction_survives_alongside_lvarchar(
conn_params: ConnParams,
) -> None:
"""The reported schema pairs FRACTION(5) columns with LVARCHARs."""
ts = datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_dt_lv "
"(s LVARCHAR(512), t DATETIME YEAR TO FRACTION(5), n INT8 NOT NULL)"
)
cur.execute(
"INSERT INTO t_dt_lv VALUES (?, ?, ?)", ("PackageRoot", ts, 10)
)
cur.execute("SELECT s, t, n FROM t_dt_lv")
assert cur.fetchone() == ("PackageRoot", ts, 10)

71
tests/test_int8_unit.py Normal file
View File

@ -0,0 +1,71 @@
"""Unit tests for the INT8 / SERIAL8 codec — no server required.
The byte vectors below were captured off the wire from both Informix
12.10.FC12W1DE and 15.0.1.0.3DE, which emit byte-identical encodings.
INT8 is sign-magnitude across two 32-bit halves, stored high-half-last:
bytes 0-1 sign word: 0 = NULL, 1 = positive, 0xFFFF = negative
bytes 2-5 LOW 32 bits, big-endian unsigned
bytes 6-9 HIGH 32 bits, big-endian unsigned
The trap: +n and -n have *identical* magnitude bytes. Anything that
treats this as a two's-complement integer is wrong for every negative
value while looking correct for every positive one.
"""
from __future__ import annotations
import pytest
from informix_db._types import IfxType
from informix_db.converters import FIXED_WIDTHS, _decode_int8
# (hex bytes, expected value) — captured from the wire.
WIRE_VECTORS = [
("0001be991a140000001c", 123456789012),
("ffffbe991a140000001c", -123456789012),
("00010000002a00000000", 42),
("ffff0000002a00000000", -42),
("00010000000000000000", 0),
("00000000000000000000", None), # sign word 0 -> NULL
("0001ffffffffffffffff", 2**64 - 1), # both halves saturated
("00010000000100000000", 1),
("00010000000000000001", 1 << 32), # high half only
]
@pytest.mark.parametrize(("hexbytes", "expected"), WIRE_VECTORS)
def test_decode_int8_wire_vectors(hexbytes: str, expected: int | None) -> None:
assert _decode_int8(bytes.fromhex(hexbytes)) == expected
def test_positive_and_negative_share_magnitude_bytes() -> None:
"""The whole reason a naive 8-byte read gets negatives wrong."""
pos = bytes.fromhex("0001be991a140000001c")
neg = bytes.fromhex("ffffbe991a140000001c")
assert pos[2:] == neg[2:], "magnitude bytes should be identical"
assert _decode_int8(pos) == -_decode_int8(neg)
def test_null_sign_word_beats_nonzero_magnitude() -> None:
"""Sign word 0 means NULL regardless of what the magnitude bytes hold."""
assert _decode_int8(bytes.fromhex("0000be991a140000001c")) is None
def test_short_payload_raises() -> None:
with pytest.raises(ValueError, match="too short"):
_decode_int8(bytes.fromhex("0001be991a14")) # 6 bytes, need 10
def test_int8_registered_as_ten_bytes() -> None:
"""A width of 8 here would desync every row containing an INT8."""
assert FIXED_WIDTHS[IfxType.INT8] == 10
assert FIXED_WIDTHS[IfxType.SERIAL8] == 10
def test_int8_width_differs_from_bigint() -> None:
"""INT8 (17) and BIGINT (52) are different types with different
widths conflating them is the easy mistake."""
assert FIXED_WIDTHS[IfxType.INT8] != FIXED_WIDTHS[IfxType.BIGINT]
assert FIXED_WIDTHS[IfxType.BIGINT] == 8

View File

@ -0,0 +1,249 @@
"""Regression tests for LVARCHAR tuple framing, reported 2026-08-31.
Two independent framing errors, both of which shifted every column that
followed an LVARCHAR:
1. **A phantom pad byte.** We appended an even-byte pad when the value
length was odd. There is no pad the next column begins immediately
after the last content byte.
2. **A missing length on NULL.** We returned as soon as the null
indicator said NULL, leaving the 4-byte length field unread. The
length is part of the envelope and is always present.
Both survived a 247-test suite because the only LVARCHAR fixture used
``'lv value'`` 8 characters, even, and never NULL. Neither faulty
branch ever executed.
The tests below therefore vary length parity and nullness deliberately,
and always place a column *after* the LVARCHAR, because the damage lands
downstream: a trailing LVARCHAR can be mis-sized with no visible effect.
Wire evidence (Informix 12.10) for INT8 / LVARCHAR / INT8:
'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes
NULL -> 01 | 00 00 00 00 5 bytes
'' -> 00 | 00 00 00 00 5 bytes
NULL and empty string differ only in the indicator byte.
"""
from __future__ import annotations
import datetime
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
# --------------------------------------------------------------------------
# Length parity — the phantom pad byte
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"text",
[
"", # 0 — even, empty
"a", # 1 — ODD
"ab", # 2
"abc", # 3 — ODD
"PackageRoot", # 11 — ODD, the reported value
"lv value", # 8 — the old fixture that hid the bug
"x" * 255, # 255 — ODD, spans a length byte boundary
"y" * 256, # 256
],
)
def test_lvarchar_length_parity_does_not_shift_next_column(
conn_params: ConnParams, text: str
) -> None:
"""A trailing sentinel column catches any over- or under-read."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_parity "
"(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL, c VARCHAR(8))"
)
cur.execute(
"INSERT INTO t_lv_parity VALUES (?, ?, ?, ?)",
(2001, text, 10, "tail"),
)
cur.execute("SELECT a, s, b, c FROM t_lv_parity")
assert cur.fetchone() == (2001, text, 10, "tail")
def test_odd_length_lvarchar_reproduces_the_report(
conn_params: ConnParams,
) -> None:
"""The exact failure shape: INT8 decoded as its own value shifted one
byte left (10 -> 2560), and the following string losing a character."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_report "
"(a INT8 NOT NULL, k LVARCHAR(512) NOT NULL,"
" b INT8 NOT NULL, v LVARCHAR(1024))"
)
cur.execute(
"INSERT INTO t_lv_report VALUES (?, ?, ?, ?)",
(2001, "PackageRoot", 10, "/content/package"),
)
cur.execute("SELECT a, k, b, v FROM t_lv_report")
row = cur.fetchone()
assert row == (2001, "PackageRoot", 10, "/content/package")
assert row[2] != 2560, "INT8 shifted one byte left"
assert row[3].startswith("/"), "leading character lost"
# --------------------------------------------------------------------------
# NULL — the missing length field
# --------------------------------------------------------------------------
def test_null_lvarchar_does_not_shift_next_column(
conn_params: ConnParams,
) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_null "
"(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL)"
)
cur.execute("INSERT INTO t_lv_null VALUES (?, ?, ?)", (3001, None, 20))
cur.execute("SELECT a, s, b FROM t_lv_null")
assert cur.fetchone() == (3001, None, 20)
def test_null_and_empty_lvarchar_are_distinguished(
conn_params: ConnParams,
) -> None:
"""They differ only in the indicator byte, so it's easy to conflate."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_ne (k INT, s LVARCHAR(64), tail INT)"
)
cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (1, None, 111))
cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (2, "", 222))
cur.execute("SELECT k, s, tail FROM t_lv_ne ORDER BY k")
assert cur.fetchall() == [(1, None, 111), (2, "", 222)]
def test_consecutive_null_lvarchars(conn_params: ConnParams) -> None:
"""Each NULL under-read by 4 bytes, so several in a row compound."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_lv_many_null "
"(a INT8 NOT NULL, s1 LVARCHAR(256), s2 LVARCHAR(256),"
" s3 LVARCHAR(256), b INT8 NOT NULL)"
)
cur.execute(
"INSERT INTO t_lv_many_null VALUES (?, ?, ?, ?, ?)",
(4001, None, None, None, 40),
)
cur.execute("SELECT a, s1, s2, s3, b FROM t_lv_many_null")
assert cur.fetchone() == (4001, None, None, None, 40)
# --------------------------------------------------------------------------
# Wide, mixed shape — several LVARCHARs interleaved with other types
# --------------------------------------------------------------------------
_WIDE_COLS = [
"tag", "key_txt", "def_txt", "cur_txt", "note_txt",
"ver", "made_by", "ident", "made_on",
]
_WIDE_ROW = (
"Gadget",
"PackageRoot", # 11 — ODD
None, # NULL
"/content/package", # 16
"z", # 1 — ODD
77,
"maker",
3001,
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
)
def _make_wide(cur) -> None:
cur.execute(
"CREATE TEMP TABLE t_lv_wide ("
" tag VARCHAR(32) NOT NULL,"
" key_txt LVARCHAR(512) NOT NULL,"
" def_txt LVARCHAR(1024),"
" cur_txt LVARCHAR(1024),"
" note_txt LVARCHAR(1024),"
" ver INT8 DEFAULT 0 NOT NULL,"
" made_by VARCHAR(100),"
" ident INT8 NOT NULL,"
" made_on DATETIME YEAR TO FRACTION(5))"
)
cur.execute(
f"INSERT INTO t_lv_wide VALUES ({', '.join(['?'] * len(_WIDE_COLS))})",
_WIDE_ROW,
)
def test_wide_mixed_row(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
_make_wide(cur)
cur.execute(f"SELECT {', '.join(_WIDE_COLS)} FROM t_lv_wide")
assert cur.fetchone() == _WIDE_ROW
@pytest.mark.parametrize("shift", range(len(_WIDE_COLS)))
def test_wide_row_survives_column_reordering(
conn_params: ConnParams, shift: int
) -> None:
"""The reporter isolated this by reordering columns — values were
correct first in the list and wrong later on. Rotating the projection
exercises every position for every type."""
with _connect(conn_params) as conn:
cur = conn.cursor()
_make_wide(cur)
order = _WIDE_COLS[shift:] + _WIDE_COLS[:shift]
cur.execute(f"SELECT {', '.join(order)} FROM t_lv_wide")
expected = tuple(_WIDE_ROW[_WIDE_COLS.index(c)] for c in order)
assert cur.fetchone() == expected
def test_select_star_wide_row(conn_params: ConnParams) -> None:
"""``SELECT *`` raised IndexError once enough LVARCHARs accumulated."""
with _connect(conn_params) as conn:
cur = conn.cursor()
_make_wide(cur)
cur.execute("SELECT * FROM t_lv_wide")
row = cur.fetchone()
names = [d[0] for d in cur.description]
assert row == tuple(_WIDE_ROW[_WIDE_COLS.index(n)] for n in names)
def test_repeated_lvarchar_columns(conn_params: ConnParams) -> None:
"""Selecting the same LVARCHAR twice doubles any per-column drift."""
with _connect(conn_params) as conn:
cur = conn.cursor()
_make_wide(cur)
cur.execute(
"SELECT ident, key_txt, ident, key_txt, ver FROM t_lv_wide"
)
assert cur.fetchone() == (3001, "PackageRoot", 3001, "PackageRoot", 77)

View File

@ -0,0 +1,76 @@
"""Guards on the package's own metadata.
The distribution is ``informix-driver``; the import module is
``informix_db``. That mismatch is a live trap: ``importlib.metadata``
needs the *distribution* name, so the string can't be derived from
``__name__``, and getting it wrong fails silently ``__version__``
degrades to the ``0.0.0+local`` sentinel instead of raising.
That is exactly what shipped between the 2026-05-08 rename and
2026.08.27: ``__init__`` still asked for ``informix-db``, so anyone who
installed the renamed package saw ``0.0.0+local``. It went unnoticed
because a stale ``informix-db`` distribution lingered in the dev
environment and answered the query.
"""
from __future__ import annotations
from pathlib import Path
import pytest
import tomllib
import informix_db
PYPROJECT = Path(__file__).resolve().parent.parent / "pyproject.toml"
@pytest.fixture(scope="module")
def pyproject() -> dict:
if not PYPROJECT.is_file():
pytest.skip("pyproject.toml not present (installed-package test run)")
with PYPROJECT.open("rb") as handle:
return tomllib.load(handle)
def test_version_is_not_the_unresolved_sentinel() -> None:
"""If this fails, the distribution name in ``__init__`` no longer
matches ``[project].name`` and every user sees a bogus version."""
assert informix_db.__version__ != "0.0.0+local", (
"__version__ fell back to its sentinel — the distribution name "
"looked up in informix_db/__init__.py does not match the installed "
"package. Check [project].name in pyproject.toml."
)
def test_version_matches_pyproject(pyproject: dict) -> None:
declared = pyproject["project"]["version"]
# PEP 440 normalisation drops leading zeros: 2026.08.27 -> 2026.8.27.
normalised = ".".join(
str(int(part)) if part.isdigit() else part
for part in declared.split(".")
)
assert informix_db.__version__ in {declared, normalised}
def test_init_looks_up_the_declared_distribution_name(pyproject: dict) -> None:
"""Catch the rename trap directly: the name passed to
``importlib.metadata.version()`` must equal ``[project].name``."""
source = (
Path(informix_db.__file__).resolve().parent / "__init__.py"
).read_text(encoding="utf-8")
declared_name = pyproject["project"]["name"]
assert f'version("{declared_name}")' in source, (
f"informix_db/__init__.py should call version({declared_name!r}) to "
"match [project].name in pyproject.toml"
)
def test_version_is_importable_and_stringy() -> None:
assert isinstance(informix_db.__version__, str)
assert informix_db.__version__
def test_public_surface_is_exported() -> None:
for name in informix_db.__all__:
assert hasattr(informix_db, name), f"{name} in __all__ but not defined"

View File

@ -0,0 +1,248 @@
"""Rewriting :N placeholders without rewriting the SQL around them.
We advertise ``paramstyle="numeric"`` to match Informix's ESQL/C
convention, and the wire protocol takes ``?``, so placeholders get
rewritten on the way out. That was ``re.sub(r":(\\d+)", "?", sql)``,
which cannot see a string literal and therefore rewrote the inside of
one::
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
stored as 'http://host?/x'
Any ``HH:MM`` time, any URL with a port, any aspect ratio, any
``key:value`` string. It wrote wrong data and said nothing.
Nothing about it was opt-in either. The rewrite runs whenever a
statement has parameters, whatever placeholder style the caller actually
used, so writing ``?`` everywhere and never touching numeric style did
not protect you. And it changed the placeholder *count* while
``num_qmarks`` was still computed from ``len(params)``, leaving driver
and server disagreeing about how many binds exist.
The lexical rules below are Informix's own, measured against 12.10,
14.10 and 15 rather than assumed from standard SQL. The one that matters
most is the backslash: ``'a\\'b'`` is an *unterminated string* to
Informix (``-282``), not an escaped quote. A scanner written to
Postgres habits would desync on it and corrupt everything after.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _rewrite_numeric_to_qmark as rewrite
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# Substitution happens where it should
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("sql", "expected"),
[
("SELECT * FROM t WHERE a = :1", "SELECT * FROM t WHERE a = ?"),
(
"SELECT * FROM t WHERE a = :1 AND b = :2",
"SELECT * FROM t WHERE a = ? AND b = ?",
),
("SELECT * FROM t WHERE a = :10", "SELECT * FROM t WHERE a = ?"),
# Already-? SQL is returned untouched, and cheaply.
("SELECT * FROM t WHERE a = ?", "SELECT * FROM t WHERE a = ?"),
("", ""),
],
)
def test_placeholders_are_rewritten(sql: str, expected: str) -> None:
assert rewrite(sql) == expected
# ---------------------------------------------------------------------------
# ...and nowhere else
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("label", "sql", "expected"),
[
(
"url-with-port",
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = :1",
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?",
),
(
"time-of-day",
"INSERT INTO log VALUES ('started at 09:15:30', :1)",
"INSERT INTO log VALUES ('started at 09:15:30', ?)",
),
(
"aspect-ratio",
"UPDATE t SET ratio = '16:9' WHERE id = :1",
"UPDATE t SET ratio = '16:9' WHERE id = ?",
),
(
"doubled-quote-escape",
"SELECT 'it''s 10:30' FROM t WHERE a = :1",
"SELECT 'it''s 10:30' FROM t WHERE a = ?",
),
(
"delimited-identifier",
'SELECT "col:1" FROM t WHERE a = :1',
'SELECT "col:1" FROM t WHERE a = ?',
),
(
"line-comment",
"-- ticket :99\nSELECT * FROM t WHERE a = :1",
"-- ticket :99\nSELECT * FROM t WHERE a = ?",
),
(
"block-comment",
"SELECT /* not :99 */ * FROM t WHERE a = :1",
"SELECT /* not :99 */ * FROM t WHERE a = ?",
),
(
"brace-comment",
"SELECT { not :99 } * FROM t WHERE a = :1",
"SELECT { not :99 } * FROM t WHERE a = ?",
),
(
"cast-operator",
"SELECT a::INT FROM t WHERE a = :1",
"SELECT a::INT FROM t WHERE a = ?",
),
(
"colon-not-a-placeholder",
"SELECT a FROM t WHERE b = ':x' AND c = :1",
"SELECT a FROM t WHERE b = ':x' AND c = ?",
),
(
"literal-after-placeholder",
"SELECT * FROM t WHERE a = :1 AND b = '10:30'",
"SELECT * FROM t WHERE a = ? AND b = '10:30'",
),
],
)
def test_quotes_and_comments_are_left_alone(
label: str, sql: str, expected: str
) -> None:
assert rewrite(sql) == expected, label
def test_backslash_does_not_escape_a_quote() -> None:
"""Informix answers -282 for ``'a\\'b'``: the backslash is an ordinary
character and the string is unterminated. A scanner that treated it
as an escape would think it was still inside the literal and stop
substituting, or worse, resume in the wrong place."""
sql = "SELECT 'a\\' FROM t WHERE x = :1"
# The quote closes at the character after the backslash, so :1 is
# outside the literal and gets substituted.
assert rewrite(sql) == "SELECT 'a\\' FROM t WHERE x = ?"
def test_unterminated_quote_substitutes_nothing_further() -> None:
"""Under-substituting leaves the server to reject SQL that was
already malformed. Guessing would corrupt a literal."""
assert rewrite("SELECT 'oops :1 FROM t") == "SELECT 'oops :1 FROM t"
def test_unterminated_block_comment_substitutes_nothing_further() -> None:
assert rewrite("SELECT /* oops :1 FROM t") == "SELECT /* oops :1 FROM t"
def test_block_comments_do_not_nest() -> None:
"""Measured: ``/* a /* b */ c */`` is a syntax error on all three
servers, so the first ``*/`` closes the comment. Treating them as
nesting would swallow live SQL."""
assert (
rewrite("SELECT /* a /* b */ :1 FROM t")
== "SELECT /* a /* b */ ? FROM t"
)
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
)
@pytest.mark.integration
def test_colon_literals_round_trip(conn_params: ConnParams) -> None:
"""The bug as a user meets it: the value stored is not the value
written. Both placeholder styles, because the rewrite ran for both."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw (id INT, s VARCHAR(60))")
cur.execute(
"INSERT INTO t_rw VALUES (?, 'http://host:8080/path')", (1,)
)
cur.execute("INSERT INTO t_rw VALUES (:1, 'at 09:15:30')", (2,))
cur.execute("SELECT id, s FROM t_rw ORDER BY id")
assert cur.fetchall() == [
(1, "http://host:8080/path"),
(2, "at 09:15:30"),
]
@pytest.mark.integration
def test_colon_literals_round_trip_through_executemany(
conn_params: ConnParams,
) -> None:
"""executemany rewrites unconditionally, so it had the bug too."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw2 (id INT, s VARCHAR(40))")
cur.executemany(
"INSERT INTO t_rw2 VALUES (?, '16:9')", [(1,), (2,)]
)
cur.execute("SELECT DISTINCT s FROM t_rw2")
assert cur.fetchall() == [("16:9",)]
@pytest.mark.integration
def test_comment_bearing_sql_still_binds(conn_params: ConnParams) -> None:
"""A leading comment reaches PREPARE now that classification asks the
server. Make sure the rewriter agrees and doesn't eat a placeholder
that follows one."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"/* report: daily */ SELECT FIRST 1 tabid FROM systables "
"WHERE tabid > :1",
(0,),
)
assert cur.fetchone() is not None
@pytest.mark.integration
def test_placeholder_count_matches_after_rewrite(
conn_params: ConnParams,
) -> None:
"""The old regex could add placeholders the driver never counted,
leaving num_qmarks (from len(params)) disagreeing with the SQL. A
literal containing three colon-digit sequences is the shape that
used to break it."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw3 (id INT, s VARCHAR(40))")
with contextlib.suppress(Exception):
cur.execute("DELETE FROM t_rw3")
cur.execute(
"INSERT INTO t_rw3 VALUES (:1, 'a:1 b:2 c:3')", (7,)
)
cur.execute("SELECT id, s FROM t_rw3")
assert cur.fetchone() == (7, "a:1 b:2 c:3")

View File

@ -0,0 +1,459 @@
"""Regression tests for error recovery, cursor lifecycle, and concurrency.
The type-matrix suite covers what a value looks like on the wire. This
covers everything around it: what happens after a statement fails, what
happens to abandoned cursors, and what happens when more than one thread
or task is involved.
Two bugs here were found by fuzzing, and both are the kind that never
show up in a single-threaded happy-path test:
* **A failed statement was never released.** Successful DML sent
PREPARE EXECUTE RELEASE; failing DML sent PREPARE EXECUTE and
stopped. The leaked statement collided with the next PREPARE, and
every subsequent call on that connection returned a nonsense error
whose offset pointed back at the *failed* SQL. A duplicate-key
violation about the most ordinary error an application can hit
killed the connection outright.
* **Cancelling a pool acquire leaked the connection.** ``asyncio.to_thread``
cannot interrupt its worker, so a cancelled waiter left the worker to
finish and hand back a connection nobody owned. Under HTTP load, where
client disconnects cancel request tasks precisely while they wait for
a connection, the pool dies one slot at a time.
The tests that follow assert on *data* wherever they can, not merely the
absence of an exception: a worker reads back a value only it wrote, so a
crossed wire fails even when nothing raises.
"""
from __future__ import annotations
import asyncio
import contextlib
import threading
import pytest
import informix_db
from informix_db import aio
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
# Bounded: a leaked statement used to manifest as a hang.
read_timeout=25.0,
**kw,
)
def _pool_kw(conn_params: ConnParams) -> dict:
return {
"host": conn_params.host,
"port": conn_params.port,
"user": conn_params.user,
"password": conn_params.password,
"database": conn_params.database,
"server": conn_params.server,
"connect_timeout": 15.0,
"read_timeout": 30.0,
}
# ---------------------------------------------------------------------------
# Error recovery — a failed statement must not poison the connection
# ---------------------------------------------------------------------------
FAILING_SQL = [
pytest.param("SELECT FROM WHERE", id="syntax"),
pytest.param("SELECT * FROM no_such_table_xyz", id="no_such_table"),
pytest.param("SELECT no_such_col FROM systables", id="no_such_column"),
pytest.param("SELECT no_such_function_xyz(1) FROM systables", id="bad_func"),
pytest.param("SELECT 1/0 FROM systables", id="div_zero"),
pytest.param("SELECT 'notanumber'::INT FROM systables", id="bad_cast"),
]
@pytest.mark.parametrize("sql", FAILING_SQL)
def test_connection_survives_failed_statement(
conn_params: ConnParams, sql: str
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
for _ in range(4): # repeat: a leak accumulates
with pytest.raises(informix_db.Error):
cur.execute(sql)
cur.fetchall()
cur.execute("SELECT FIRST 1 tabid FROM systables ORDER BY tabid")
assert cur.fetchone() is not None
@pytest.mark.parametrize("autocommit", [True, False])
def test_duplicate_key_does_not_kill_the_connection(
conn_params: ConnParams, autocommit: bool
) -> None:
"""The bug that started this file. A unique-constraint violation left
the prepared statement allocated server-side; the next statement then
failed with a nonsense error (-255 "Not in transaction" in autocommit,
-285 otherwise) and the connection never recovered."""
with _connect(conn_params, autocommit=autocommit) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_dup (k INT PRIMARY KEY)")
cur.execute("INSERT INTO t_dup VALUES (1)")
if not autocommit:
conn.commit()
for i in range(5):
with pytest.raises(informix_db.IntegrityError):
cur.execute("INSERT INTO t_dup VALUES (1)")
if not autocommit:
conn.rollback()
cur.execute("SELECT k FROM t_dup")
assert cur.fetchall() == [(1,)], f"broken after {i + 1} violations"
def test_failed_dml_with_params_releases_statement(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_nn (s VARCHAR(8) NOT NULL, k INT)")
for _ in range(4):
with pytest.raises(informix_db.Error):
cur.execute("INSERT INTO t_nn VALUES (?, ?)", (None, 1))
cur.execute("SELECT COUNT(*) FROM t_nn")
assert cur.fetchone() == (0,)
def test_new_cursor_works_after_error(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.Error):
cur.execute("SELECT * FROM no_such_table_xyz")
other = conn.cursor()
other.execute("SELECT FIRST 1 tabid FROM systables")
assert other.fetchone() is not None
other.close()
# ---------------------------------------------------------------------------
# Fetch batching — NFETCH is a byte budget, so batch edges move with width
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [0, 1, 2, 3, 63, 64, 65, 199, 200, 201, 512, 1025])
def test_row_count_exact_across_fetch_styles(
conn_params: ConnParams, n: int
) -> None:
"""Every fetch style must agree, at counts that straddle plausible
server batch boundaries. A batching off-by-one shows up as a wrong
count, a duplicate, or a dropped row."""
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_batch "
"(k INT, a VARCHAR(32), b INT, c VARCHAR(32), d INT)"
)
if n:
cur.executemany(
"INSERT INTO t_batch VALUES (?, ?, ?, ?, ?)",
[(i, f"a{i}", i * 2, f"c{i}", i * 3) for i in range(n)],
)
want = [(i,) for i in range(n)]
sql = "SELECT k FROM t_batch ORDER BY k"
cur.execute(sql)
assert cur.fetchall() == want
cur.execute(sql)
got = []
while (row := cur.fetchone()) is not None:
got.append(row)
assert got == want
for size in (1, 7, 100):
cur.execute(sql)
got = []
while chunk := cur.fetchmany(size):
got.extend(chunk)
assert got == want, f"fetchmany({size}) disagreed at n={n}"
cur.execute(sql)
assert list(cur) == want
# ---------------------------------------------------------------------------
# Cursor lifecycle
# ---------------------------------------------------------------------------
def test_abandoned_partial_fetches_do_not_leak(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_aband (k INT)")
cur.executemany(
"INSERT INTO t_aband VALUES (?)", [(i,) for i in range(200)]
)
for i in range(25):
c = conn.cursor()
c.execute("SELECT k FROM t_aband ORDER BY k")
c.fetchone()
if i % 2:
c.close()
else:
del c # rely on the finalizer
cur.execute("SELECT COUNT(*) FROM t_aband")
assert cur.fetchone() == (200,)
def test_re_execute_mid_fetch(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_reex (k INT)")
cur.executemany(
"INSERT INTO t_reex VALUES (?)", [(i,) for i in range(100)]
)
for _ in range(10):
cur.execute("SELECT k FROM t_reex ORDER BY k")
cur.fetchone() # abandon mid-fetch
cur.execute("SELECT k FROM t_reex ORDER BY k")
assert len(cur.fetchall()) == 100
def test_interleaved_cursors_on_one_connection(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_inter (k INT)")
cur.executemany(
"INSERT INTO t_inter VALUES (?)", [(i,) for i in range(50)]
)
a, b = conn.cursor(), conn.cursor()
a.execute("SELECT k FROM t_inter ORDER BY k")
b.execute("SELECT k FROM t_inter ORDER BY k DESC")
assert [a.fetchone() for _ in range(5)] == [(i,) for i in range(5)]
assert [b.fetchone() for _ in range(5)] == [
(i,) for i in range(49, 44, -1)
]
a.close()
b.close()
# ---------------------------------------------------------------------------
# Concurrency
# ---------------------------------------------------------------------------
def test_threads_sharing_one_connection_do_not_interleave(
conn_params: ConnParams,
) -> None:
"""One connection is one socket. Each thread reads back a value only
it supplied, so crossed wires fail even if nothing raises."""
failures: list[str] = []
lock = threading.Lock()
with _connect(conn_params, autocommit=True) as conn:
def worker(tid: int) -> None:
try:
for r in range(10):
cur = conn.cursor()
token = tid * 1000 + r
cur.execute(
"SELECT FIRST 1 ?::INT, ?::VARCHAR(16) FROM systables",
(token, f"t{tid}"),
)
row = cur.fetchone()
if row != (token, f"t{tid}"):
with lock:
failures.append(f"thread {tid}: got {row!r}")
return
cur.close()
except Exception as exc:
with lock:
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
for t in threads:
t.start()
for t in threads:
t.join()
assert not failures, failures
def test_pool_hands_out_clean_connections_under_load(
conn_params: ConnParams,
) -> None:
"""Borrowers that fail must still return a usable connection."""
failures: list[str] = []
lock = threading.Lock()
pool = informix_db.create_pool(
**_pool_kw(conn_params), min_size=2, max_size=4,
acquire_timeout=30.0, autocommit=True,
)
try:
def worker(tid: int) -> None:
try:
for r in range(8):
with pool.connection() as conn:
cur = conn.cursor()
if r % 3 == 0:
with contextlib.suppress(informix_db.Error):
cur.execute("SELECT * FROM no_such_tbl_xyz")
token = tid * 1000 + r
cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (token,)
)
got = cur.fetchone()
if got != (token,):
with lock:
failures.append(
f"thread {tid}: got {got!r}, want {token}"
)
return
cur.close()
except Exception as exc:
with lock:
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
for t in threads:
t.start()
for t in threads:
t.join()
finally:
with contextlib.suppress(Exception):
pool.close()
assert not failures, failures
def test_pool_does_not_leak_transactions_between_borrowers(
conn_params: ConnParams,
) -> None:
"""max_size=1 so both borrowers get the same underlying connection."""
pool = informix_db.create_pool(
**_pool_kw(conn_params), min_size=1, max_size=1,
acquire_timeout=30.0, autocommit=False,
)
try:
with pool.connection() as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_pool_txn")
cur.execute("CREATE TABLE t_pool_txn (k INT)")
conn.commit()
try:
for i in range(5):
with pool.connection() as conn: # dirties, never commits
conn.cursor().execute(
"INSERT INTO t_pool_txn VALUES (?)", (i,)
)
with pool.connection() as conn: # must not see it
cur = conn.cursor()
cur.execute("SELECT COUNT(*) FROM t_pool_txn")
assert cur.fetchone() == (0,), "transaction leaked"
conn.rollback()
finally:
with pool.connection() as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_pool_txn")
conn.commit()
finally:
with contextlib.suppress(Exception):
pool.close()
# ---------------------------------------------------------------------------
# Async
# ---------------------------------------------------------------------------
async def test_async_pool_survives_cancelled_acquires(
conn_params: ConnParams,
) -> None:
"""Cancelling a task blocked on acquire used to destroy that pool slot
permanently: ``asyncio.to_thread`` cannot interrupt its worker, so the
worker finished and handed back a connection nobody owned.
Two holders keep both connections checked out; three waiters block on
acquire and are cancelled there. Afterwards the pool must still serve.
"""
pool = await aio.create_pool(
**_pool_kw(conn_params), min_size=1, max_size=2, autocommit=True
)
try:
release = asyncio.Event()
async def holder(i: int) -> None:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (i,)
)
await cur.fetchone()
await release.wait()
async def waiter(i: int) -> tuple | None:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (i,)
)
return await cur.fetchone()
for cycle in range(2):
release.clear()
holders = [asyncio.create_task(holder(i)) for i in range(2)]
await asyncio.sleep(0.5) # both connections held
waiters = [asyncio.create_task(waiter(100 + i)) for i in range(3)]
await asyncio.sleep(0.5) # all blocked on acquire
for t in waiters:
t.cancel()
await asyncio.gather(*waiters, return_exceptions=True)
release.set()
await asyncio.gather(*holders, return_exceptions=True)
await asyncio.sleep(0.3) # orphan returns land
got = await asyncio.wait_for(waiter(200 + cycle), timeout=15)
assert got == (200 + cycle,), (
f"pool starved after cycle {cycle} — cancelled acquires leaked"
)
finally:
with contextlib.suppress(Exception):
await pool.close()
async def test_async_pool_concurrent_tasks_get_their_own_data(
conn_params: ConnParams,
) -> None:
pool = await aio.create_pool(
**_pool_kw(conn_params), min_size=2, max_size=4, autocommit=True
)
try:
async def worker(tid: int) -> None:
for r in range(6):
async with pool.connection() as conn:
cur = await conn.cursor()
token = tid * 1000 + r
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (token,)
)
assert await cur.fetchone() == (token,)
await asyncio.gather(*(worker(i) for i in range(6)))
finally:
with contextlib.suppress(Exception):
await pool.close()

View File

@ -0,0 +1,230 @@
"""End-of-row reconciliation, and the two bugs it found immediately.
Fourteen framing bugs reached users before this check existed. Every one
was the same defect a column consuming the wrong number of bytes and
not one raised at the point of the mistake. The wrong width produced a
plausible value and corrupted whatever came *next*, so the damage always
surfaced in a different column, row, or statement.
All fourteen were detectable for free. ``payload`` is a fully extracted
``bytes`` object of known length, so after decoding N columns the offset
must land exactly on the end. ``_row_not_consumed`` raises when it
doesn't, naming the column shape and the byte delta.
Turning the check on found two more bugs on its first run against the
existing suite:
* **Smart LOBs were read as a flat 72-byte field.** They use the UDT
envelope: 149 bytes populated (``[ind=0][len=144][144 hex chars]``),
5 bytes NULL. We consumed 72 and left 77 behind, so any LOB not in the
final column position corrupted everything after it. The 144 bytes are
the 72-byte locator *hex-encoded*, which means ``BlobLocator.raw`` had
never actually held a locator only the first half of the hex text.
Nobody noticed because ``read_blob_column`` resolves LOBs through the
server-side ``lotofile`` function and never uses the locator.
* **NULL composite UDTs skipped their length field**, byte-for-byte the
LVARCHAR NULL bug in the branch twelve lines away.
Both were the same ``[indicator][int32 length][data]`` envelope, written
out by hand a third and fourth time. ``_read_udt_envelope`` is now the
single implementation.
"""
from __future__ import annotations
import contextlib
import io
import struct
import pytest
import informix_db
from informix_db._protocol import IfxStreamReader, ProtocolError
from informix_db._resultset import (
ColumnInfo,
_read_udt_envelope,
parse_tuple_payload,
)
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The check itself — unit level, no server
# ---------------------------------------------------------------------------
def _tuple_pdu(body: bytes) -> IfxStreamReader:
"""Wrap a payload in the SQ_TUPLE framing parse_tuple_payload expects."""
return IfxStreamReader(
io.BytesIO(struct.pack("!h", 0) + struct.pack("!i", len(body)) + body)
)
def test_under_read_is_detected() -> None:
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
with pytest.raises(ProtocolError, match="under-read by 6"):
parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5) + b"LEFTOV"), cols)
def test_exact_consumption_passes() -> None:
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
assert parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5)), cols) == (5,)
def test_error_names_the_column_shape() -> None:
""""row decode failed" is not actionable. The message has to say which
shape and by how much, so the next report arrives with the answer."""
cols = [
ColumnInfo(name="ident", type_code=2, raw_type_code=2, encoded_length=4),
ColumnInfo(name="qty", type_code=2, raw_type_code=2, encoded_length=4),
]
body = struct.pack("!i", 1) + struct.pack("!i", 2) + b"UNCONSUMED"
with pytest.raises(ProtocolError) as exc:
parse_tuple_payload(_tuple_pdu(body), cols)
msg = str(exc.value)
assert "ident" in msg and "qty" in msg, "message must name the columns"
assert "tc=2" in msg
assert "consumed 8 of 18" in msg, "message must give the byte counts"
assert "under-read by 10" in msg
# ---------------------------------------------------------------------------
# The shared UDT envelope
# ---------------------------------------------------------------------------
def test_envelope_reads_length_even_when_null() -> None:
"""The length belongs to the envelope, not the value. Returning early
on the indicator is the bug that hit LVARCHAR and then composites."""
payload = bytes([1]) + struct.pack("!i", 0) + b"NEXT"
offset, body = _read_udt_envelope(payload, 0)
assert body is None
assert offset == 5, "NULL envelope must still consume its length field"
assert payload[offset:] == b"NEXT"
def test_envelope_reads_body_when_present() -> None:
payload = bytes([0]) + struct.pack("!i", 3) + b"abc" + b"NEXT"
offset, body = _read_udt_envelope(payload, 0)
assert body == b"abc"
assert offset == 8
assert payload[offset:] == b"NEXT"
def test_envelope_rejects_negative_length() -> None:
"""A negative length off the wire would rewind the offset and silently
re-decode earlier bytes as a later column."""
payload = bytes([0]) + struct.pack("!i", -4) + b"abc"
with pytest.raises(ProtocolError, match="negative length"):
_read_udt_envelope(payload, 0)
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
pytestmark_integration = pytest.mark.integration
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host, port=conn_params.port, user=conn_params.user,
password=conn_params.password, database=conn_params.database,
server=conn_params.server, connect_timeout=15.0, read_timeout=30.0,
autocommit=True,
)
@pytest.mark.integration
def test_null_composite_does_not_shift_next_column(
conn_params: ConnParams,
) -> None:
"""A NULL SET made the following VARCHAR return '' instead of its value."""
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_nullset")
cur.execute(
"CREATE TABLE t_nullset (a INT, s SET(INT NOT NULL), b VARCHAR(10))"
)
try:
cur.execute("INSERT INTO t_nullset VALUES (7, SET{1,2}, 'abc')")
cur.execute("INSERT INTO t_nullset VALUES (8, NULL, 'xyz')")
cur.execute("SELECT a, s, b FROM t_nullset ORDER BY a")
rows = cur.fetchall()
assert rows[0][0] == 7 and rows[0][2] == "abc"
assert rows[1] == (8, None, "xyz"), (
"NULL composite shifted the following column"
)
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_nullset")
@pytest.mark.integration
@pytest.mark.parametrize("populated", [False, True])
def test_lob_in_non_final_position(
conn_params: ConnParams, populated: bool
) -> None:
"""A LOB was read as 72 bytes when it occupies 149 (or 5 when NULL), so
anything selected after it was garbage."""
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_lobmid")
try:
cur.execute(
"CREATE TABLE t_lobmid (a INT, b BLOB, c INT, d VARCHAR(10))"
)
except informix_db.Error:
pytest.skip("no sbspace configured; see make ifx-spaces")
try:
if populated:
cur.write_blob_column(
"INSERT INTO t_lobmid VALUES (?, BLOB_PLACEHOLDER, ?, ?)",
b"hello", (1, 222222, "tail"),
)
else:
cur.execute(
"INSERT INTO t_lobmid VALUES (1, NULL, 222222, 'tail')"
)
cur.execute("SELECT a, b, c, d FROM t_lobmid")
a, b, c, d = cur.fetchone()
assert (a, c, d) == (1, 222222, "tail"), (
"columns after the LOB were shifted"
)
if populated:
assert isinstance(b, informix_db.BlobLocator)
assert len(b.raw) == 72, (
"locator must be the decoded 72 bytes, not hex text"
)
else:
assert b is None
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_lobmid")
@pytest.mark.integration
def test_wide_mixed_row_reconciles(conn_params: ConnParams) -> None:
"""Belt and braces: a row using most of the tricky types must consume
its payload exactly. The check runs on every fetch, so merely getting
a row back proves reconciliation held."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_recon ("
" a INT8 NOT NULL, k LVARCHAR(512), n NCHAR(8), v VARCHAR(32),"
" d DECIMAL(16), b BOOLEAN, t DATETIME YEAR TO FRACTION(5),"
" tail INT)"
)
cur.execute(
"INSERT INTO t_recon VALUES (?, ?, ?, ?, ?, ?, CURRENT, ?)",
(1, "PackageRoot", "nch", "v", None, True, 424242),
)
cur.execute(
"SELECT a, k, n, v, d, b, t, tail FROM t_recon"
)
row = cur.fetchone()
assert row[0] == 1
assert row[-1] == 424242

435
tests/test_rows.py Normal file
View File

@ -0,0 +1,435 @@
"""Named row access, and the cost of it.
A field request: `row[11]` tells a reader nothing on a wide projection,
and `pyodbc` / `mssql-python` both hand back rows that answer to
position, column name, and attribute at once. `row_factory=Row` gives
the same three.
It is opt-in. Defaulting it on would tax the thing this driver is
measured against, since supporting `row["name"]` means `__getitem__`
becomes a Python method rather than C-level tuple indexing, which costs
roughly 39 ns on every subscript. Users who want readable column access
in application code should pay that; a bulk export that never looks at a
column by name should not.
`Row` subclasses `tuple`, so `row == (1, "x")` still holds and nothing
that already worked stops working. That constraint drove the design: the
existing suite compares fetched rows against plain tuples in hundreds of
places, and so does everybody's code.
"""
from __future__ import annotations
import contextlib
import pickle
import pytest
import informix_db
from informix_db.rows import _RESERVED, Row, make_row_class
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The type itself
# ---------------------------------------------------------------------------
def _row(fields, values):
return make_row_class(tuple(fields))(values)
def test_three_ways_to_reach_a_column() -> None:
row = _row(("tabid", "tabname"), (1, "systables"))
assert row[0] == 1
assert row["tabname"] == "systables"
assert row.tabname == "systables"
def test_still_equal_to_a_plain_tuple() -> None:
"""The compatibility constraint. Existing code and the existing test
suite compare fetched rows against tuples everywhere."""
row = _row(("a", "b"), (1, "x"))
plain = (1, "x")
assert row == plain
# Both directions: tuple.__eq__ on the left has to accept a subclass
# on the right, or `expected == fetched` assertions break.
assert plain == row
assert list(row) == [1, "x"]
assert len(row) == 2
a, b = row
assert (a, b) == (1, "x")
assert row in [(1, "x")]
def test_slice_degrades_to_a_plain_tuple() -> None:
"""A slice has no meaningful column mapping, so it should not pretend
to be a Row."""
row = _row(("a", "b", "c"), (1, 2, 3))
assert row[0:2] == (1, 2)
assert type(row[0:2]) is tuple
def test_negative_index_still_works() -> None:
assert _row(("a", "b"), (1, 2))[-1] == 2
def test_keys_and_asdict() -> None:
row = _row(("a", "b"), (1, "x"))
assert row.keys() == ("a", "b")
assert row._asdict() == {"a": 1, "b": "x"}
def test_repr_names_the_columns() -> None:
assert repr(_row(("a", "b"), (1, "x"))) == "Row(a=1, b='x')"
def test_pickles() -> None:
"""The per-shape class is built at runtime, so it cannot be pickled by
reference. Multiprocessing users would hit that immediately."""
row = _row(("a", "b"), (1, "x"))
restored = pickle.loads(pickle.dumps(row))
assert restored == (1, "x")
assert restored.b == "x"
def test_missing_column_says_what_is_there() -> None:
row = _row(("tabid", "tabname"), (1, "x"))
with pytest.raises(KeyError, match="tabid"):
_ = row["nope"]
with pytest.raises(AttributeError, match="tabid"):
_ = row.nope
def test_class_is_cached_per_shape() -> None:
"""A type() call per execute() would land on every small query."""
assert make_row_class(("a", "b")) is make_row_class(("a", "b"))
assert make_row_class(("a", "b")) is not make_row_class(("a", "c"))
def test_duplicate_names_resolve_to_the_first() -> None:
"""``SELECT tabid, tabid`` is legal and both columns are named
``tabid``. pyodbc gives the first; so do we."""
row = _row(("tabid", "tabid"), (1, 2))
assert row["tabid"] == 1
assert row.tabid == 1
assert row[1] == 2, "positional access must still reach the second"
@pytest.mark.parametrize("name", sorted(_RESERVED))
def test_a_column_always_beats_a_method_of_the_same_name(name: str) -> None:
"""Parametrized over the *computed* reserved set rather than a
hand-written list, because the hand-written list is what missed
``keys``, ``_asdict`` and ``_fields`` on the first attempt. Adding a
method to Row later cannot silently reopen the hole: this test grows
with it.
Without the guard these return a bound method, which is the exact
failure shape this driver spent a fortnight removing from its
decoders: a plausible-looking wrong answer, in silence."""
row = _row((name, "other"), (7, 9))
assert getattr(row, name) == 7
assert row[name] == 7
def test_shadowed_methods_stay_reachable_through_the_base_class() -> None:
row = _row(("count", "keys", "_asdict"), (1, 2, 3))
assert tuple.count(row, 1) == 1
assert Row.keys(row) == ("count", "keys", "_asdict")
assert Row._asdict(row) == {"count": 1, "keys": 2, "_asdict": 3}
def test_shadowing_fields_does_not_break_repr_or_asdict() -> None:
"""The machinery reads its own field list, so if a column named
``_fields`` shadowed it, repr and _asdict would report the column
value instead of the names. Mangled attributes keep them separate."""
row = _row(("_fields", "x"), ("not the names", 2))
assert row._fields == "not the names"
assert Row.keys(row) == ("_fields", "x")
assert repr(row) == "Row(_fields='not the names', x=2)"
def test_non_identifier_names_are_subscript_only() -> None:
"""Informix names expression columns things like ``(count(*))``,
which cannot be an attribute."""
row = _row(("(count(*))", "ok"), (5, 1))
assert row["(count(*))"] == 5
assert row.ok == 1
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
**kw,
)
@pytest.mark.integration
def test_default_is_still_a_plain_tuple(conn_params: ConnParams) -> None:
"""The opt-in has to be genuinely opt-in. Anyone who does not ask for
Row should not pay for it or see any behaviour change."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 1 tabid, tabname FROM systables")
row = cur.fetchone()
assert type(row) is tuple
with pytest.raises(TypeError):
_ = row["tabname"]
@pytest.mark.integration
def test_row_factory_on_the_connection(conn_params: ConnParams) -> None:
"""One line at connect time, then every cursor from it, which is what
'without any additional coding' has to mean in practice."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
cur.execute(
"SELECT FIRST 1 tabid, tabname FROM systables ORDER BY tabid"
)
row = cur.fetchone()
assert row[0] == row["tabid"] == row.tabid
assert row[1] == row["tabname"] == row.tabname
@pytest.mark.integration
def test_every_fetch_path_returns_rows(conn_params: ConnParams) -> None:
"""fetchone, fetchmany, fetchall and iteration each return rows by a
different route through the cursor."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
sql = "SELECT FIRST 4 tabid, tabname FROM systables ORDER BY tabid"
cur = conn.cursor()
cur.execute(sql)
assert cur.fetchone().tabname is not None
cur.execute(sql)
assert all(r.tabname is not None for r in cur.fetchmany(2))
cur.execute(sql)
assert all(r.tabname is not None for r in cur.fetchall())
cur.execute(sql)
assert all(r.tabname is not None for r in cur)
@pytest.mark.integration
def test_scrollable_cursor_returns_rows(conn_params: ConnParams) -> None:
"""Scrollable cursors return each row straight from the wire rather
than from the materialized list, so they are a separate path."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT tabid, tabname FROM systables ORDER BY tabid")
assert cur.fetch_first().tabname is not None
assert cur.fetch_absolute(1).tabname is not None
cur.close()
@pytest.mark.integration
def test_per_cursor_override(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
named = conn.cursor()
named.row_factory = informix_db.Row
plain = conn.cursor()
sql = "SELECT FIRST 1 tabid FROM systables"
named.execute(sql)
plain.execute(sql)
assert isinstance(named.fetchone(), Row)
assert type(plain.fetchone()) is tuple
@pytest.mark.integration
def test_informix_lowercases_so_the_obvious_name_works(
conn_params: ConnParams,
) -> None:
"""Informix folds unquoted identifiers, which is why the .lower() in
the workaround people write by hand is a no-op."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rows (Config_Key INT, Cnt INT)")
cur.execute("INSERT INTO t_rows VALUES (1, 2)")
cur.execute("SELECT Config_Key, Cnt FROM t_rows")
row = cur.fetchone()
assert row.config_key == 1
assert row.cnt == 2
@pytest.mark.integration
def test_pool_forwards_the_factory(conn_params: ConnParams) -> None:
pool = informix_db.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
row_factory=informix_db.Row,
min_size=1,
max_size=2,
)
try:
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone().tabid is not None
finally:
pool.close()
def test_zero_column_row() -> None:
row = make_row_class(())(())
assert row == ()
assert row.keys() == ()
assert repr(row) == "Row()"
def test_wide_row() -> None:
"""500 columns: the name map is a dict, so this should be flat, but
it is the shape most likely to expose an off-by-one."""
fields = tuple(f"c{i}" for i in range(500))
row = make_row_class(fields)(tuple(range(500)))
assert row["c499"] == row.c499 == row[499] == 499
assert row["c0"] == row[0] == 0
def test_names_that_cannot_be_attributes_are_subscript_only() -> None:
row = _row(("col with space", "1leading_digit", ""), (1, 2, 3))
assert row["col with space"] == 1
assert row["1leading_digit"] == 2
assert row[""] == 3
def test_unicode_column_name() -> None:
row = _row(("café", "x"), (1, 2))
assert row["café"] == 1
assert row.café == 1
def test_dunder_named_column_is_subscript_only() -> None:
"""A dunder cannot be shadowed without breaking the object protocol,
so the column is reachable by subscript and the attribute keeps its
ordinary meaning. No real schema should notice."""
row = _row(("__class__", "ok"), (1, 2))
assert row["__class__"] == 1
assert row.__class__.__name__ == "Row"
def test_row_outlives_eviction_of_its_class_from_the_cache() -> None:
"""The class cache is bounded. A row already handed to the caller
holds its class alive, so eviction must not affect it."""
row = make_row_class(("z1", "z2"))((9, 8))
for i in range(300):
make_row_class((f"evict{i}", "b"))
assert (row.z1, row["z2"], row[0]) == (9, 8, 9)
def test_class_cache_is_bounded() -> None:
for i in range(400):
make_row_class((f"bounded{i}", "b"))
assert make_row_class.cache_info().currsize <= 256
# ---------------------------------------------------------------------------
# Row must not change a single value
# ---------------------------------------------------------------------------
_WIDE_DDL = """CREATE TABLE t_rowdiff (
c_int INT, c_small SMALLINT, c_big BIGINT, c_int8 INT8, c_serial SERIAL,
c_float FLOAT, c_smallfloat SMALLFLOAT, c_dec DECIMAL(16,4),
c_decu DECIMAL(16), c_money MONEY(12,2), c_char CHAR(10),
c_vchar VARCHAR(40), c_nchar NCHAR(8), c_lvar LVARCHAR(200), c_date DATE,
c_dt DATETIME YEAR TO FRACTION(5), c_ivl INTERVAL YEAR TO MONTH,
c_bool BOOLEAN, c_set SET(INT NOT NULL), c_tail INT)"""
_WIDE_COLS = (
"c_int,c_small,c_big,c_int8,c_serial,c_float,c_smallfloat,c_dec,c_decu,"
"c_money,c_char,c_vchar,c_nchar,c_lvar,c_date,c_dt,c_ivl,c_bool,c_set,"
"c_tail"
)
@pytest.mark.integration
def test_rows_are_value_identical_to_tuples(conn_params: ConnParams) -> None:
"""The strongest statement available: across every awkward type, with
a fully populated row and a fully NULL one, wrapping must change
nothing. If it does, Row is not a presentation layer, it is a bug."""
sql = f"SELECT {_WIDE_COLS} FROM t_rowdiff ORDER BY c_tail"
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_rowdiff")
cur.execute(_WIDE_DDL)
try:
cur.execute(
"INSERT INTO t_rowdiff VALUES (1,2,3,4,0,1.5,2.5,12.34,99,"
"9.99,'ch','vc','nc','lv',TODAY,CURRENT,"
"INTERVAL(1-2) YEAR TO MONTH,'t',SET{1,2},77)"
)
cur.execute(
"INSERT INTO t_rowdiff VALUES (NULL,NULL,NULL,NULL,0,NULL,"
"NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,"
"NULL,NULL,88)"
)
cur.execute(sql)
plain = cur.fetchall()
with _connect(conn_params, row_factory=informix_db.Row) as named_conn:
named = named_conn.cursor()
named.execute(sql)
rows = named.fetchall()
assert rows == plain, "wrapping changed a value"
names = _WIDE_COLS.split(",")
first = rows[0]
for i, name in enumerate(names):
assert first[i] == first[name] == getattr(first, name), name
assert rows[1]["c_int"] is None
assert rows[1].c_tail == 88
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_rowdiff")
@pytest.mark.integration
def test_threads_share_one_class_per_shape(conn_params: ConnParams) -> None:
"""The cache is process-wide and the pool hands connections to many
threads, so two threads running the same query must land on the same
class rather than racing to build competing ones."""
import threading
seen: list[type] = []
errors: list[Exception] = []
def worker() -> None:
try:
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
for _ in range(5):
cur.execute(
"SELECT FIRST 1 tabid, tabname FROM systables"
)
row = cur.fetchone()
assert row.tabname == row["tabname"] == row[1]
seen.append(type(row))
except Exception as exc:
errors.append(exc)
threads = [threading.Thread(target=worker) for _ in range(6)]
for t in threads:
t.start()
for t in threads:
t.join(60)
assert not errors, errors[:2]
assert len({id(c) for c in seen}) == 1, "same shape built more than once"

172
tests/test_socket_reads.py Normal file
View File

@ -0,0 +1,172 @@
"""Two readers, one stream, and a length field taken on trust.
``IfxSocket`` owns a read-ahead buffer that ``BufferedSocketReader``
fills and drains. But ``Connection._drain_to_eot``, ``_raise_sq_err``
and the login path bypass that reader and call ``IfxSocket.read_exact``
directly, which recv'd from the socket without ever looking at the
buffer. Bytes sitting in the buffer would simply be skipped, and skipped
bytes in a length-framed protocol don't announce themselves — the next
read lands mid-field and every read after it is wrong.
Nothing triggers it today. The server sends one response per request, so
recv returns exactly that response and the buffered reader consumes all
of it before control returns to a direct read. That is a property of the
traffic, not of the code, and the buffer is connection-scoped precisely
so read-ahead *can* cross response boundaries pipelined executemany
already puts several responses in flight. A latent desync waiting on a
timing change is not a good thing to leave in a wire protocol.
Separately, ``fill_recv_buf`` took its byte count on trust, and that
count is almost always a length field straight off the wire. A corrupt
or desynced stream turned into an allocation of whatever the field
happened to say: a garbage ``0x7FFFFFFF`` reads as a 2 GB request, and
the fill loop sits in recv until the read timeout while the buffer
grows. The limit turns that into an error that names the number, which
is the actual diagnostic a length that absurd means framing was
already lost upstream.
"""
from __future__ import annotations
import pytest
from informix_db._protocol import BufferedSocketReader, ProtocolError
from informix_db._socket import MAX_READ_BYTES, IfxSocket
class _FakeSocket:
"""Stands in for the raw socket. Records what recv actually asked for."""
def __init__(self, data: bytes = b"") -> None:
self.data = data
self.pos = 0
self.recv_calls: list[int] = []
def recv(self, n: int) -> bytes:
self.recv_calls.append(n)
chunk = self.data[self.pos : self.pos + n]
self.pos += len(chunk)
return chunk
def close(self) -> None:
# The EOF path force-closes; a stand-in has to survive that.
pass
def _socket_with(buffered: bytes, on_wire: bytes = b"") -> IfxSocket:
"""An IfxSocket with ``buffered`` already read ahead into _recv_buf."""
sock = IfxSocket.__new__(IfxSocket)
sock._sock = _FakeSocket(on_wire)
sock._recv_buf = bytearray(buffered)
sock._recv_pos = 0
sock._recv_size = 65536
sock._read_timeout = None
return sock
# ---------------------------------------------------------------------------
# read_exact must not step over the buffer
# ---------------------------------------------------------------------------
def test_read_exact_consumes_the_buffer_first() -> None:
sock = _socket_with(b"BUFFERED", on_wire=b"SOCKET")
assert sock.read_exact(8) == b"BUFFERED"
assert sock._sock.recv_calls == [], "must not touch the socket at all"
assert sock._recv_pos == 8
def test_read_exact_spans_buffer_then_socket() -> None:
"""The interesting case: a read that starts in the buffer and
finishes on the wire. Getting this wrong reorders the stream."""
sock = _socket_with(b"HEAD", on_wire=b"TAIL")
assert sock.read_exact(8) == b"HEADTAIL"
assert sock._sock.recv_calls == [4], "only the shortfall comes from recv"
def test_read_exact_respects_a_partly_consumed_buffer() -> None:
sock = _socket_with(b"XXABCD")
sock._recv_pos = 2 # first two bytes already decoded
assert sock.read_exact(4) == b"ABCD"
assert sock._sock.recv_calls == []
def test_read_exact_of_zero_is_empty() -> None:
sock = _socket_with(b"DATA")
assert sock.read_exact(0) == b""
assert sock.read_exact(-5) == b"", "a negative count must not rewind"
assert sock._recv_pos == 0
def test_short_read_error_reports_the_original_request() -> None:
"""The message counts bytes; taking some from the buffer must not make
it lie about how many were asked for."""
from informix_db.exceptions import OperationalError
sock = _socket_with(b"AB", on_wire=b"") # 2 buffered, nothing on the wire
with pytest.raises(OperationalError, match="wanted 10 bytes"):
sock.read_exact(10)
def test_buffered_reader_and_direct_read_agree_on_one_stream() -> None:
"""End to end: a BufferedSocketReader over-reads, then a direct
read_exact picks up exactly where it left off."""
sock = _socket_with(b"", on_wire=b"\x00\x2aREST-OF-THE-STREAM")
reader = BufferedSocketReader(sock)
assert reader.read_short() == 42
assert len(sock._recv_buf) - sock._recv_pos > 0, (
"precondition: the reader must have over-read for this to mean "
"anything"
)
assert sock.read_exact(18) == b"REST-OF-THE-STREAM"
# ---------------------------------------------------------------------------
# fill_recv_buf must not believe an arbitrary length
# ---------------------------------------------------------------------------
def test_absurd_length_is_refused_not_allocated() -> None:
sock = _socket_with(b"", on_wire=b"")
with pytest.raises(ProtocolError, match="refusing to read"):
sock.fill_recv_buf(MAX_READ_BYTES + 1)
assert sock._sock.recv_calls == [], "must refuse before any recv"
def test_refusal_names_the_knob() -> None:
"""The error has to be actionable in both directions: framing is lost,
or the value genuinely is that big and the limit needs raising."""
sock = _socket_with(b"", on_wire=b"")
with pytest.raises(ProtocolError) as exc:
sock.fill_recv_buf(2**31 - 1)
message = str(exc.value)
assert "2147483647" in message
assert "IFX_MAX_READ_BYTES" in message
def test_a_normal_length_is_unaffected() -> None:
sock = _socket_with(b"", on_wire=b"x" * 100)
sock.fill_recv_buf(100)
assert len(sock._recv_buf) - sock._recv_pos >= 100
# ---------------------------------------------------------------------------
# skip
# ---------------------------------------------------------------------------
def test_buffered_skip_does_not_rewind_on_a_negative_count() -> None:
"""The base reader's skip delegates to read_exact, which guards. This
one advances the cursor arithmetically, so an unguarded negative count
re-decodes bytes already consumed as if they were the next field."""
sock = _socket_with(b"ABCDEFGH")
sock._recv_pos = 4
BufferedSocketReader(sock).skip(-4)
assert sock._recv_pos == 4, "skip must never move the cursor backwards"
def test_buffered_skip_advances_normally() -> None:
sock = _socket_with(b"ABCDEFGH")
reader = BufferedSocketReader(sock)
reader.skip(4)
assert reader.read_exact(4) == b"EFGH"

View File

@ -0,0 +1,216 @@
"""Deciding whether a statement needs a cursor, by asking rather than guessing.
The driver chose between "open a cursor and fetch" and "execute and
release" by checking whether the first word of the SQL was ``SELECT``.
That gets five ordinary forms wrong a leading comment in any of the
three Informix flavours, a parenthesized select, a parenthesized UNION,
and a CTE. All five are perfectly good queries, and all five failed with
``-260 Cursor name already in use``, an error that describes neither the
cause nor anything the caller did. It says "cursor" because the driver
sent SQ_EXECUTE where the server was waiting to open one.
The server had been telling us the answer the whole time.
``statement_type`` is the first field of the DESCRIBE response, and
``parse_describe`` has always parsed it into the metadata dict, where
nothing read it. Every SELECT form above reports 2.
The comment that justified the heuristic said ``nfields`` couldn't
distinguish these cases, because ``INSERT INTO t VALUES (?)`` also
describes a column. That much was true and it argued for the wrong
conclusion, because ``statement_type`` distinguishes them exactly. That
INSERT reports 6.
The predicate is now JDBC's ``IfxSqli.isResultSet``: type 2, or type 56
(``EXECUTE PROCEDURE``/``FUNCTION``) with at least one column, since a
routine may or may not return rows. That last clause fixes
``EXECUTE FUNCTION`` as a side effect it used to run down the DML path
and discard its return value.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _produces_result_set
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The predicate
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("statement_type", "ncolumns", "expected", "why"),
[
(2, 1, True, "SELECT"),
(2, 0, True, "SELECT describing no columns is still a SELECT"),
(6, 1, False, "INSERT ... VALUES (?) describes a column but is DML"),
(6, 0, False, "INSERT with literals"),
(32, 0, False, "DELETE"),
(33, 0, False, "UPDATE"),
(45, 0, False, "CREATE"),
(56, 0, False, "EXECUTE PROCEDURE returning nothing"),
(56, 2, True, "EXECUTE FUNCTION returning rows"),
(0, 0, False, "unknown type defaults to the non-cursor path"),
],
)
def test_result_set_predicate(
statement_type: int, ncolumns: int, expected: bool, why: str
) -> None:
assert _produces_result_set(statement_type, ncolumns) is expected, why
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
)
_ONE = "SELECT FIRST 1 tabid FROM systables"
@pytest.mark.integration
@pytest.mark.parametrize(
("label", "sql"),
[
pytest.param("plain", _ONE, id="plain"),
pytest.param("lowercase", _ONE.lower(), id="lowercase"),
pytest.param("leading-whitespace", f" \n\t {_ONE}", id="whitespace"),
pytest.param("line-comment", f"-- pick one\n{_ONE}", id="line-comment"),
pytest.param("block-comment", f"/* pick one */ {_ONE}", id="block-comment"),
pytest.param("brace-comment", f"{{ pick one }} {_ONE}", id="brace-comment"),
pytest.param(
"cte",
"WITH c AS (SELECT tabid FROM systables) "
"SELECT FIRST 1 tabid FROM c",
id="cte",
),
pytest.param("parenthesized", f"({_ONE})", id="parenthesized"),
pytest.param(
"union", f"{_ONE} UNION SELECT 99 FROM systables", id="union"
),
pytest.param(
"parenthesized-union",
f"({_ONE}) UNION (SELECT 99 FROM systables)",
id="paren-union",
),
],
)
def test_every_select_form_opens_a_cursor(
conn_params: ConnParams, label: str, sql: str
) -> None:
"""The five non-``plain`` forms below the whitespace case all failed
with -260 under the first-word heuristic."""
with _connect(conn_params) as conn:
cur = conn.cursor()
try:
cur.execute(sql)
except informix_db.ProgrammingError as exc:
# Informix 12.10 has no CTEs and rejects WITH at offset 1.
# A syntax error is the server declining the grammar, which
# is a different thing from the driver routing it wrongly —
# that produced -260, not -201.
if getattr(exc, "sqlcode", None) == -201:
pytest.skip(f"server does not support this syntax: {label}")
raise
rows = cur.fetchall()
assert rows, f"{label}: expected rows, got none"
assert cur.description is not None
@pytest.mark.integration
def test_parameterized_insert_is_not_mistaken_for_a_query(
conn_params: ConnParams,
) -> None:
"""The case the old comment worried about, and the reason it kept the
heuristic: this DESCRIBEs a column. ``statement_type`` says 6."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_cls (k INT)")
cur.execute("INSERT INTO t_cls VALUES (?)", (7,))
assert cur.rowcount == 1
cur.execute("SELECT k FROM t_cls")
assert cur.fetchall() == [(7,)]
@pytest.mark.integration
def test_execute_function_returns_its_value(conn_params: ConnParams) -> None:
"""Type 56 with columns. Under the first-word heuristic this ran down
the DML path and the return value was discarded."""
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP FUNCTION ifxdrv_dbl")
cur.execute(
"CREATE FUNCTION ifxdrv_dbl(n INT) RETURNING INT; "
"RETURN n * 2; END FUNCTION"
)
try:
cur.execute("EXECUTE FUNCTION ifxdrv_dbl(21)")
assert cur.fetchall() == [(42,)]
finally:
with contextlib.suppress(Exception):
cur.execute("DROP FUNCTION ifxdrv_dbl")
@pytest.mark.integration
def test_dml_still_takes_the_non_cursor_path(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_cls2 (k INT)")
cur.execute("INSERT INTO t_cls2 VALUES (1)")
cur.execute("UPDATE t_cls2 SET k = 2")
assert cur.rowcount == 1
cur.execute("DELETE FROM t_cls2")
assert cur.rowcount == 1
assert cur.description is None
@pytest.mark.integration
def test_executemany_refuses_a_query_the_first_word_missed(
conn_params: ConnParams,
) -> None:
"""The pre-flight check shares the heuristic's blind spots, so a
comment-prefixed SELECT reaches PREPARE. The post-DESCRIBE check
catches it, and the connection stays usable the refusal releases
the statement. (A leading comment rather than a CTE, so this also
runs on 12.10, which has no CTEs.)"""
with _connect(conn_params) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.NotSupportedError):
cur.executemany(
"/* batched? no */ SELECT FIRST 1 tabid FROM systables "
"WHERE tabid <> ?",
[(1,), (2,)],
)
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, "refusal leaked the statement"
@pytest.mark.integration
def test_executemany_still_refuses_a_plain_select(
conn_params: ConnParams,
) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.NotSupportedError):
cur.executemany(
"SELECT FIRST 1 tabid FROM systables WHERE tabid <> ?",
[(1,), (2,)],
)

View File

@ -0,0 +1,189 @@
"""Every door out of a statement has to release it on the way through.
A statement that fails is still allocated server-side. Skipping the
RELEASE bricks the connection: the next PREPARE collides with the leaked
one, and every subsequent call returns a nonsense error whose offset
points back at the *failed* SQL rather than the new statement.
There are six exits from the execute paths, and the guard was added to
them one at a time, each after a user hit it:
1. ``_execute_dml`` drain
2. ``_execute_dml_with_params`` build
3. ``_execute_dml_with_params`` drain
4. ``executemany`` pipeline build/send
5. ``_execute_select_with_params`` build
6. ``_execute_select`` fetch loop
Two were still open, and both are ordinary to reach:
* **The parameterized-SELECT bind drain.** The build was guarded and the
drain was not. Passing a string where the column is an INT gets a
clean encode, so the rejection comes from the *server* (-1213, -415)
during the bind drain past the guard. A wrong-typed parameter is
about as common as application mistakes get.
* **The scrollable-cursor open.** No guard at all, and the worst place
to lack one: the GC-time finalizer is armed on the line *after* the
drain, so a failure there left the statement allocated with no
fallback of any kind.
The guard is now one helper rather than six hand-rolled copies, which
also fixed a defect in the copies that had a cursor to close: CLOSE and
RELEASE shared a single ``contextlib.suppress`` block, so a CLOSE that
raised skipped the RELEASE losing the half that actually matters.
"""
from __future__ import annotations
import pytest
import informix_db
from informix_db.cursors import _RELEASE_PDU, Cursor
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The helper — no server needed
# ---------------------------------------------------------------------------
class _FakeConn:
def __init__(self) -> None:
self.sent: list[bytes] = []
def _send_pdu(self, pdu: bytes) -> None:
self.sent.append(pdu)
class _FakeCursor:
"""Duck-types the four attributes ``_release_after_failure`` touches."""
def __init__(self, *, close_raises: bool = False) -> None:
self._conn = _FakeConn()
self._close_raises = close_raises
def _build_close_pdu(self) -> bytes:
return b"CLOSE"
def _build_release_pdu(self) -> bytes:
return _RELEASE_PDU
def _drain_to_eot(self) -> None:
if self._close_raises and self._conn.sent[-1] == b"CLOSE":
raise OSError("wire went away mid-close")
def test_release_is_sent_even_when_close_fails() -> None:
"""The regression in the hand-rolled copies. CLOSE and RELEASE shared
one suppress block, so a failing CLOSE swallowed the RELEASE and a
lost cursor handle is a nuisance while a lost statement breaks the
next call."""
cur = _FakeCursor(close_raises=True)
Cursor._release_after_failure(cur, close_cursor=True)
assert _RELEASE_PDU in cur._conn.sent, (
"a CLOSE that raises must not prevent the RELEASE"
)
def test_close_is_skipped_when_no_cursor_was_opened() -> None:
cur = _FakeCursor()
Cursor._release_after_failure(cur)
assert cur._conn.sent == [_RELEASE_PDU]
def test_cleanup_never_propagates() -> None:
"""Cleanup runs inside an ``except``. If it raises, it replaces the
real SQL error with a secondary failure from the cleanup path, which
is strictly worse for the caller."""
class _Hostile(_FakeCursor):
def _drain_to_eot(self) -> None:
raise OSError("connection reset")
Cursor._release_after_failure(_Hostile(), close_cursor=True)
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
# Bounded: a leaked statement used to manifest as a hang.
read_timeout=25.0,
autocommit=True,
)
@pytest.mark.integration
@pytest.mark.parametrize(
("sql", "params"),
[
pytest.param(
"SELECT tabid FROM systables WHERE tabid = ?",
("not-an-int",),
id="string-for-int",
),
pytest.param(
"SELECT ?::INT FROM systables WHERE tabid = 1",
(2**40,),
id="out-of-range-int",
),
],
)
def test_server_rejected_bind_releases_statement(
conn_params: ConnParams, sql: str, params: tuple
) -> None:
"""A parameterized SELECT whose bind the *server* rejects. The value
encodes cleanly, so this lands past the build guard and in the drain
that had none. Repeated because a leak only shows on the call after
it the first failure looks fine on its own."""
with _connect(conn_params) as conn:
cur = conn.cursor()
for i in range(4):
with pytest.raises(informix_db.Error):
cur.execute(sql, params)
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, f"broken after {i + 1} binds"
@pytest.mark.integration
def test_failed_scroll_open_releases_statement(conn_params: ConnParams) -> None:
"""FOR UPDATE prepares cleanly and fails at OPEN with -526, which is
exactly the branch that had no guard."""
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
for i in range(4):
with pytest.raises(informix_db.Error):
scroll.execute("SELECT tabid FROM systables FOR UPDATE")
assert not scroll._server_cursor_open, (
"a failed open must not leave the cursor marked live"
)
other = conn.cursor()
other.execute("SELECT FIRST 1 tabid FROM systables")
assert other.fetchone() is not None, f"broken after {i + 1} opens"
other.close()
scroll.close()
@pytest.mark.integration
def test_scrollable_cursor_still_works_after_a_failed_open(
conn_params: ConnParams,
) -> None:
"""The same cursor object must be reusable — a failed open is an
ordinary error, not a terminal state for the cursor."""
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
with pytest.raises(informix_db.Error):
scroll.execute("SELECT tabid FROM systables FOR UPDATE")
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
assert scroll.fetch_first() is not None
scroll.close()

View File

@ -0,0 +1,355 @@
"""One statement per session, and what happens when we forget that.
SQLI gives a session a single statement slot. ``SQ_CLOSE``,
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on whatever statement is
current none of them names one. Ordinary use never notices, because a
non-scrollable cursor materializes its rows and releases the statement
before returning, so the slot is free again by the time anyone looks.
A scrollable cursor is the exception: it holds the slot open on purpose.
Two things then went wrong, and neither said so.
**Another statement on the same connection.** The server returns ``-285``
for the new statement and *also* destroys the scrollable cursor its
next fetch comes back ``-267`` "the transaction has been rolled back,
all locks released". Two unattributable errors from code that reads as
completely ordinary: iterate a large result set, run a lookup partway
through. It is now a ``ProgrammingError`` that says what happened.
**The deferred-cleanup queue drained at the wrong moment.** A cursor
finalizer that can't get the wire lock hands its CLOSE/RELEASE to a
queue for the next operation to flush. That queue was flushed before
*every* PDU. But a finalizer enqueues precisely because another thread
holds the lock, i.e. is mid-statement so the flush landed inside that
thread's own statement and released it. The victim saw ``-208`` when it
happened before the first fetch, ``-267`` between fetch batches. The
flush now happens only at a statement boundary, where the queued CLOSE
addresses the orphan it was meant for.
A stale queue entry was fatal too: the finalizer enqueues, the cursor
then gets closed properly, and the leftover CLOSE draws ``-267`` an
``OperationalError``, which is in ``WIRE_ERRORS``, which force-closed a
perfectly healthy connection. Server-reported errors are now told apart
from wire failures by the presence of a ``sqlcode``.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _CLOSE_PDU, _RELEASE_PDU
from tests.conftest import ConnParams
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
**kw,
)
# ---------------------------------------------------------------------------
# The conflict check — bookkeeping, no server
# ---------------------------------------------------------------------------
class _FakeScroll:
def __init__(self, open_: bool = True) -> None:
self._server_cursor_open = open_
def _conflict_check(conn: informix_db.Connection, requester: object) -> None:
conn._check_scroll_cursor_conflict(requester)
def test_conflict_check_is_silent_with_no_scroll_cursor() -> None:
conn = informix_db.Connection.__new__(informix_db.Connection)
conn._open_scroll_cursor = None
_conflict_check(conn, object())
def test_conflict_check_forgets_a_collected_cursor() -> None:
"""The ref is weak so an abandoned scrollable cursor can still be
finalized. A dead ref means the slot is free."""
import weakref
conn = informix_db.Connection.__new__(informix_db.Connection)
victim = _FakeScroll()
conn._open_scroll_cursor = weakref.ref(victim)
del victim
_conflict_check(conn, object())
assert conn._open_scroll_cursor is None
def test_conflict_check_lets_the_owner_through() -> None:
"""Re-executing the *same* scrollable cursor is allowed — it closes
its own server-side cursor first."""
import weakref
conn = informix_db.Connection.__new__(informix_db.Connection)
owner = _FakeScroll()
conn._open_scroll_cursor = weakref.ref(owner)
_conflict_check(conn, owner)
def test_conflict_check_forgets_a_closed_cursor() -> None:
import weakref
conn = informix_db.Connection.__new__(informix_db.Connection)
done = _FakeScroll(open_=False)
conn._open_scroll_cursor = weakref.ref(done)
_conflict_check(conn, object())
assert conn._open_scroll_cursor is None
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
@pytest.mark.integration
def test_second_statement_is_refused_while_scroll_cursor_is_open(
conn_params: ConnParams,
) -> None:
"""Used to be -285 for the new statement plus -267 for the scrollable
cursor. Now it's one error that names the cause, and the scrollable
cursor is untouched."""
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
first = scroll.fetch_first()
assert first is not None
other = conn.cursor()
with pytest.raises(informix_db.ProgrammingError, match="scrollable"):
other.execute("SELECT COUNT(*) FROM systables")
assert scroll.fetch_absolute(1) is not None, (
"the refused statement must not have disturbed the cursor"
)
scroll.close()
other.execute("SELECT COUNT(*) FROM systables")
assert other.fetchone() is not None, "slot must free on close"
@pytest.mark.integration
def test_executemany_is_refused_too(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_slot (k INT)")
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
scroll.fetch_first()
with pytest.raises(informix_db.ProgrammingError, match="scrollable"):
cur.executemany("INSERT INTO t_slot VALUES (?)", [(1,), (2,)])
scroll.close()
cur.executemany("INSERT INTO t_slot VALUES (?)", [(1,), (2,)])
cur.execute("SELECT COUNT(*) FROM t_slot")
assert cur.fetchone() == (2,)
@pytest.mark.integration
def test_scroll_cursor_can_be_re_executed(conn_params: ConnParams) -> None:
"""The owner is the one caller allowed past the conflict check, and
that is only safe because it closes its own server-side cursor
first. Without that it collides with itself."""
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
for _ in range(4):
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
assert scroll.fetch_first() is not None
scroll.close()
@pytest.mark.integration
def test_abandoning_a_scroll_cursor_frees_the_slot(
conn_params: ConnParams,
) -> None:
"""Dropping the last reference must let the connection be used again
the finalizer closes the cursor and the weak ref goes dead."""
import gc
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
scroll.fetch_first()
del scroll
gc.collect()
cur = conn.cursor()
cur.execute("SELECT COUNT(*) FROM systables")
assert cur.fetchone() is not None
# ---------------------------------------------------------------------------
# Deferred cleanup
# ---------------------------------------------------------------------------
@pytest.mark.integration
def test_queued_cleanup_does_not_land_inside_a_running_statement(
conn_params: ConnParams,
) -> None:
"""Exactly what a cross-thread finalizer does: it lost the wire lock,
so it queued its CLOSE/RELEASE while another thread was mid-statement.
The flush used to happen before that thread's very next PDU — its own
CURNAME/NFETCH releasing the statement out from under it."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_defer (k INT)")
cur.executemany(
"INSERT INTO t_defer VALUES (?)", [(i,) for i in range(50)]
)
original = cur._read_describe_response
def enqueue_between_prepare_and_fetch() -> None:
result = original()
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
return result
cur._read_describe_response = enqueue_between_prepare_and_fetch
try:
cur.execute("SELECT k FROM t_defer ORDER BY k")
assert len(cur.fetchall()) == 50, (
"the queued cleanup released our own statement"
)
finally:
cur._read_describe_response = original
@pytest.mark.integration
def test_stale_queued_cleanup_does_not_kill_the_connection(
conn_params: ConnParams,
) -> None:
"""A queue entry goes stale whenever the cursor gets closed properly
between enqueue and flush. The server answers the leftover CLOSE with
-267, which is an OperationalError, which is in WIRE_ERRORS so a
stale entry used to force-close a healthy connection."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 1 tabid FROM systables")
cur.fetchall()
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, "stale cleanup killed the connection"
assert not conn.closed
assert conn._pending_cleanup == [], "queue should have drained"
@pytest.mark.integration
def test_connection_closes_cleanly_with_a_scroll_cursor_open(
conn_params: ConnParams,
) -> None:
conn = _connect(conn_params)
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
scroll.fetch_first()
with contextlib.suppress(Exception):
conn.close()
assert conn.closed
# ---------------------------------------------------------------------------
# The wire lock's blind spot
# ---------------------------------------------------------------------------
def test_wire_lock_reports_its_own_thread() -> None:
"""The whole point. ``RLock.acquire(blocking=False)`` returns True for
the owning thread, so the finalizer's probe could not tell "nobody is
using the wire" from "I am, right now, mid-statement"."""
from informix_db.connections import _WireLock
lock = _WireLock()
assert not lock.held_by_current_thread
with lock:
assert lock.held_by_current_thread
assert lock.acquire(blocking=False), "must still be reentrant"
lock.release()
assert lock.held_by_current_thread, "still held at depth 1"
assert not lock.held_by_current_thread
def test_wire_lock_is_not_held_by_other_threads() -> None:
import threading
from informix_db.connections import _WireLock
lock = _WireLock()
seen: list[bool] = []
entered = threading.Event()
done = threading.Event()
def holder() -> None:
with lock:
entered.set()
done.wait(5)
t = threading.Thread(target=holder)
t.start()
entered.wait(5)
seen.append(lock.held_by_current_thread)
done.set()
t.join(5)
assert seen == [False], "another thread's hold must not read as ours"
@pytest.mark.integration
def test_finalizer_defers_when_gc_runs_on_the_locking_thread(
conn_params: ConnParams,
) -> None:
"""GC fires on whatever thread allocated. When that is the thread
holding the wire lock, the finalizer must write nothing it used to
acquire the RLock reentrantly and send CLOSE/RELEASE into the running
statement, killing it with -208."""
import gc
with _connect(conn_params) as conn:
gc.disable()
try:
victim = conn.cursor(scrollable=True)
victim.execute("SELECT tabid FROM systables ORDER BY tabid")
victim.fetch_first()
# A reference cycle, so collection waits for gc rather than
# happening at the drop. Cycles are ordinary in Python.
cycle = [victim]
cycle.append(cycle)
del victim, cycle
writes: list[bytes] = []
original_write = conn._sock.write_all
conn._sock.write_all = lambda b: (
writes.append(b),
original_write(b),
)[1]
try:
with conn._wire_lock: # stand in for "mid-statement"
gc.collect()
assert writes == [], (
"finalizer wrote to the wire while a statement "
"owned it"
)
assert conn._pending_cleanup, (
"cleanup should have been deferred, not dropped"
)
finally:
conn._sock.write_all = original_write
finally:
gc.enable()
cur = conn.cursor()
cur.execute("SELECT COUNT(*) FROM systables")
assert cur.fetchone() is not None

333
tests/test_tls_traffic.py Normal file
View File

@ -0,0 +1,333 @@
"""End-to-end SQLI traffic over a real TLS socket.
``tests/test_tls.py`` covers the handshake. This covers what happens
*after* it: every byte of real SQLI traffic crossing genuine TLS records,
through the same codecs, reader, and cursor machinery the plain-socket
suite exercises.
That distinction matters because ``SSLSocket.recv`` is not
``socket.recv``. It returns at most one TLS record's worth of plaintext
however much you ask for, it can return fewer bytes than are available,
and plaintext buffered inside the SSL object is invisible to the OS. The
Phase 39 buffered reader asks for up to 64 KB per call and loops until
satisfied that loop is the thing which has to be right, and nothing in
the plain-socket suite puts the same pressure on it.
**Scope.** A TLS-terminating proxy in front of the plain SQLI listener
supplies the TLS half. This tests the driver's TLS path, which is the
half we own. It does *not* test IBM's server-side TLS listener: Informix
15 wants a PKCS#12 keystore whose stash the developer-edition image
rejects (``GSK_ERROR_BAD_KEYFILE_PASSWORD``), and that side is IBM's
code. Anything below ``ssl.wrap_socket`` is identical either way.
Skipped when ``openssl`` isn't on PATH — the proxy needs a certificate.
"""
from __future__ import annotations
import contextlib
import datetime
import decimal
import select
import shutil
import socket
import ssl
import subprocess
import tempfile
import threading
from pathlib import Path
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
# ---------------------------------------------------------------------------
# TLS-terminating proxy
# ---------------------------------------------------------------------------
class _TlsProxy:
"""Accepts TLS, relays plaintext to the real Informix listener."""
def __init__(self, backend: tuple[str, int]) -> None:
self.backend = backend
self.tmpdir = tempfile.mkdtemp(prefix="ifx-tls-test-")
self.cert = str(Path(self.tmpdir) / "cert.pem")
key = str(Path(self.tmpdir) / "key.pem")
subprocess.run(
["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes",
"-keyout", key, "-out", self.cert, "-days", "1",
"-subj", "/CN=127.0.0.1"],
check=True, capture_output=True,
)
self._ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
self._ctx.load_cert_chain(self.cert, key)
self._sock = socket.socket()
self._sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._sock.bind(("127.0.0.1", 0))
self._sock.listen(64)
self.port: int = self._sock.getsockname()[1]
self._stop = threading.Event()
threading.Thread(target=self._serve, daemon=True).start()
def _serve(self) -> None:
self._sock.settimeout(0.5)
while not self._stop.is_set():
try:
raw, _ = self._sock.accept()
except (TimeoutError, OSError):
continue
threading.Thread(
target=self._handle, args=(raw,), daemon=True
).start()
def _handle(self, raw: socket.socket) -> None:
try:
client = self._ctx.wrap_socket(raw, server_side=True)
except (ssl.SSLError, OSError):
with contextlib.suppress(OSError):
raw.close()
return
try:
upstream = socket.create_connection(self.backend, timeout=20)
except OSError:
with contextlib.suppress(OSError):
client.close()
return
try:
self._pump(client, upstream)
finally:
for s in (client, upstream):
with contextlib.suppress(OSError):
s.close()
@staticmethod
def _pump(a: socket.socket, b: socket.socket) -> None:
# Drain the SSL object's own buffer before consulting select():
# select only sees the OS socket, so already-decrypted bytes
# sitting inside the SSL object would stall the relay.
socks = [a, b]
while True:
pending = [s for s in socks
if isinstance(s, ssl.SSLSocket) and s.pending()]
ready = pending or select.select(socks, [], [], 1.0)[0]
for s in ready:
other = b if s is a else a
try:
data = s.recv(65536)
except (ssl.SSLError, OSError):
return
if not data:
return
try:
other.sendall(data)
except OSError:
return
def close(self) -> None:
self._stop.set()
with contextlib.suppress(OSError):
self._sock.close()
shutil.rmtree(self.tmpdir, ignore_errors=True)
@pytest.fixture(scope="module")
def tls_proxy(conn_params: ConnParams):
if shutil.which("openssl") is None:
pytest.skip("openssl not on PATH; needed to generate a test cert")
proxy = _TlsProxy((conn_params.host, conn_params.port))
try:
yield proxy
finally:
proxy.close()
def _client_ctx(proxy: _TlsProxy) -> ssl.SSLContext:
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.load_verify_locations(proxy.cert)
ctx.check_hostname = False # self-signed CN=127.0.0.1
return ctx
def _connect(proxy: _TlsProxy, conn_params: ConnParams, **kw):
return informix_db.connect(
host="127.0.0.1",
port=proxy.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=20.0,
read_timeout=45.0,
tls=_client_ctx(proxy),
**kw,
)
# ---------------------------------------------------------------------------
# Traffic
# ---------------------------------------------------------------------------
def test_query_over_tls(tls_proxy, conn_params: ConnParams) -> None:
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 3 tabname FROM systables ORDER BY tabid")
assert len(cur.fetchall()) == 3
assert conn.server_version, "server_version empty over TLS"
def test_type_round_trip_over_tls(tls_proxy, conn_params: ConnParams) -> None:
"""The types that gave us framing bugs, every byte through TLS."""
ts = datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
row = (
2001, "PackageRoot", None, "/content/package", 77,
decimal.Decimal("1234567890123456"), True, ts, "nch",
)
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_tls_types ("
" a INT8 NOT NULL, k LVARCHAR(512), d LVARCHAR(512),"
" v LVARCHAR(1024), n INT8, dec16 DECIMAL(16), b BOOLEAN,"
" t DATETIME YEAR TO FRACTION(5), c NCHAR(6))"
)
cur.execute(
"INSERT INTO t_tls_types VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", row
)
cur.execute("SELECT a, k, d, v, n, dec16, b, t, c FROM t_tls_types")
assert cur.fetchone() == row
@pytest.mark.parametrize("size", [1, 4096, 16383, 16384, 16385, 32000])
def test_payload_spans_tls_record_boundary(
tls_proxy, conn_params: ConnParams, size: int
) -> None:
"""A TLS record holds ~16 KB, so these straddle the boundary where a
single ``recv`` stops being enough."""
payload = "x" * size
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_tls_big (k INT, v LVARCHAR(32000))")
cur.execute("INSERT INTO t_tls_big VALUES (?, ?)", (1, payload))
cur.execute("SELECT v, k FROM t_tls_big")
assert cur.fetchone() == (payload, 1)
@pytest.mark.parametrize("n", [500, 5000])
def test_bulk_fetch_over_tls(
tls_proxy, conn_params: ConnParams, n: int
) -> None:
"""Total bytes far beyond both one TLS record and the reader's 64 KB
recv budget, so the top-up loop runs many times."""
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_tls_bulk (k INT, v VARCHAR(240))")
cur.executemany(
"INSERT INTO t_tls_bulk VALUES (?, ?)",
[(i, f"row{i}-" + "y" * 200) for i in range(n)],
)
cur.execute("SELECT k, v FROM t_tls_bulk ORDER BY k")
rows = cur.fetchall()
assert len(rows) == n
assert rows[0][0] == 0
assert rows[-1][0] == n - 1
def test_error_recovery_over_tls(tls_proxy, conn_params: ConnParams) -> None:
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_tls_dup (k INT PRIMARY KEY)")
cur.execute("INSERT INTO t_tls_dup VALUES (1)")
for _ in range(4):
with pytest.raises(informix_db.Error):
cur.execute("INSERT INTO t_tls_dup VALUES (1)")
with pytest.raises(informix_db.Error):
cur.execute("SELECT * FROM t_tls_no_such_table_xyz")
cur.execute("SELECT k FROM t_tls_dup")
assert cur.fetchall() == [(1,)]
def test_concurrent_tls_connections(tls_proxy, conn_params: ConnParams) -> None:
"""Separate TLS sessions must not cross data."""
failures: list[str] = []
lock = threading.Lock()
def worker(tid: int) -> None:
tag = f"t{tid}-{'z' * 12}"
try:
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
for r in range(6):
token = tid * 1000 + r
cur.execute(
"SELECT FIRST 1 ?::INT, ?::VARCHAR(24) FROM systables",
(token, tag),
)
if cur.fetchone() != (token, tag):
with lock:
failures.append(f"thread {tid}: crossed data")
return
except Exception as exc:
with lock:
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(4)]
for t in threads:
t.start()
for t in threads:
t.join()
assert not failures, failures
# ---------------------------------------------------------------------------
# Negative cases — misuse must fail cleanly, never hang or downgrade
# ---------------------------------------------------------------------------
def test_tls_client_against_plaintext_port_fails(
tls_proxy, conn_params: ConnParams
) -> None:
with pytest.raises(informix_db.Error):
informix_db.connect(
host=conn_params.host, port=conn_params.port,
user=conn_params.user, password=conn_params.password,
database=conn_params.database, server=conn_params.server,
connect_timeout=10.0, read_timeout=10.0,
tls=_client_ctx(tls_proxy),
)
def test_plaintext_client_against_tls_port_fails(
tls_proxy, conn_params: ConnParams
) -> None:
"""Must raise rather than hang — a stalled handshake is the failure
mode that looks like a dead application."""
with pytest.raises(informix_db.Error):
informix_db.connect(
host="127.0.0.1", port=tls_proxy.port,
user=conn_params.user, password=conn_params.password,
database=conn_params.database, server=conn_params.server,
connect_timeout=10.0, read_timeout=10.0,
)
def test_verification_rejects_self_signed(
tls_proxy, conn_params: ConnParams
) -> None:
"""`tls=True` disables verification by design; a caller-supplied
verifying context must still reject an untrusted cert."""
strict = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
strict.check_hostname = True
strict.verify_mode = ssl.CERT_REQUIRED
with pytest.raises(informix_db.Error):
informix_db.connect(
host="127.0.0.1", port=tls_proxy.port,
user=conn_params.user, password=conn_params.password,
database=conn_params.database, server=conn_params.server,
connect_timeout=10.0, read_timeout=10.0, tls=strict,
)

View File

@ -0,0 +1,191 @@
"""Transaction control run as SQL, and the flag that didn't notice.
``Connection._in_transaction`` decides whether ``commit()`` and
``rollback()`` send anything at all, and the pool reads it to decide
whether a returned connection needs cleaning up. It was maintained
solely by the driver's own implicit ``SQ_BEGIN``, so a caller who wrote
``cursor.execute("BEGIN WORK")`` an entirely reasonable thing to
write walked straight past it.
With autocommit on, nothing stopped that statement reaching the server.
A transaction opened, the flag stayed False, and ``rollback()`` returned
successfully having sent nothing. The rows it was asked to discard were
still there. The connection then went back to the pool holding an open
transaction and its locks, because the pool's cleanup is guarded by the
same flag.
With autocommit off it failed instead, and for a sillier reason: the
driver's implicit ``SQ_BEGIN`` fired first, so the caller's ``BEGIN
WORK`` got ``-535``, "already in transaction". The driver and the user
competing to open the same transaction, and the user losing.
The server labels these statements: type 34 for BEGIN, 35 for COMMIT, 36
for ROLLBACK, with and without the ``WORK`` keyword. JDBC reads the same
three values off the describe and calls ``setTxBeginState`` /
``setTxEndState``. It also calls ``initiateTransaction`` *after* the
describe rather than before, which is what makes the skip possible
until the describe lands you can't know the statement is transaction
control.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _TX_CONTROL_TYPES
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
**kw,
)
def test_transaction_control_types_are_what_the_server_says(
logged_db_params: ConnParams,
) -> None:
"""Pin the three constants against a live server rather than trusting
the decompiled source. Both spellings must map to the same type."""
with _connect(logged_db_params, autocommit=True) as conn:
cur = conn.cursor()
seen = {}
for sql in ("BEGIN WORK", "COMMIT WORK", "ROLLBACK WORK",
"BEGIN", "COMMIT", "ROLLBACK"):
with conn._wire_lock:
conn._send_pdu(
cur._build_prepare_pdu(sql, num_qmarks=0),
statement_boundary=True,
)
cur._read_describe_response()
cur._release_after_failure()
seen[sql] = cur._statement_type
assert seen["BEGIN WORK"] == seen["BEGIN"] == 34
assert seen["COMMIT WORK"] == seen["COMMIT"] == 35
assert seen["ROLLBACK WORK"] == seen["ROLLBACK"] == 36
assert set(seen.values()) == _TX_CONTROL_TYPES
def test_rollback_after_sql_begin_actually_rolls_back(
logged_db_params: ConnParams,
) -> None:
"""The data-loss case. rollback() reported success and sent nothing,
so the row it was asked to discard survived."""
with _connect(logged_db_params, autocommit=True) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate")
cur.execute("CREATE TABLE t_txstate (k INT)")
try:
cur.execute("BEGIN WORK")
assert conn._in_transaction, "SQL BEGIN must set the flag"
cur.execute("INSERT INTO t_txstate VALUES (1)")
conn.rollback()
cur.execute("SELECT COUNT(*) FROM t_txstate")
assert cur.fetchone() == (0,), "rollback() was a silent no-op"
assert not conn._in_transaction
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate")
def test_sql_commit_clears_the_flag(logged_db_params: ConnParams) -> None:
"""The mirror image: with the flag stuck True after a SQL COMMIT, the
next rollback() would send SQ_RBWORK with no transaction open and
draw -255."""
with _connect(logged_db_params, autocommit=True) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate2")
cur.execute("CREATE TABLE t_txstate2 (k INT)")
try:
cur.execute("BEGIN WORK")
cur.execute("INSERT INTO t_txstate2 VALUES (1)")
cur.execute("COMMIT WORK")
assert not conn._in_transaction, "SQL COMMIT must clear the flag"
conn.rollback() # must be a no-op, not a -255
cur.execute("SELECT COUNT(*) FROM t_txstate2")
assert cur.fetchone() == (1,), "the committed row must survive"
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate2")
def test_sql_begin_does_not_collide_with_the_implicit_one(
logged_db_params: ConnParams,
) -> None:
"""Non-autocommit. The driver's implicit SQ_BEGIN used to fire first
and the caller's BEGIN WORK then got -535."""
with _connect(logged_db_params, autocommit=False) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate3")
conn.commit()
cur.execute("CREATE TABLE t_txstate3 (k INT)")
conn.commit()
try:
cur.execute("BEGIN WORK")
cur.execute("INSERT INTO t_txstate3 VALUES (1)")
conn.rollback()
cur.execute("SELECT COUNT(*) FROM t_txstate3")
assert cur.fetchone() == (0,)
conn.commit()
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate3")
conn.commit()
def test_ordinary_dml_still_opens_a_transaction(
logged_db_params: ConnParams,
) -> None:
"""_ensure_transaction moved from before the PREPARE to after the
describe. It still has to fire for everything that isn't transaction
control, or non-autocommit DML runs outside a transaction."""
with _connect(logged_db_params, autocommit=False) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate4")
conn.commit()
cur.execute("CREATE TABLE t_txstate4 (k INT)")
conn.commit()
try:
assert not conn._in_transaction
cur.execute("INSERT INTO t_txstate4 VALUES (1)")
assert conn._in_transaction, "DML must open a transaction"
conn.rollback()
cur.execute("SELECT COUNT(*) FROM t_txstate4")
assert cur.fetchone() == (0,)
conn.commit()
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate4")
conn.commit()
def test_a_failed_transaction_statement_does_not_move_the_flag(
logged_db_params: ConnParams,
) -> None:
"""The sync runs on the success path only. A COMMIT that the server
rejects has not ended anything."""
with _connect(logged_db_params, autocommit=True) as conn:
cur = conn.cursor()
assert not conn._in_transaction
with pytest.raises(informix_db.Error):
cur.execute("COMMIT WORK") # -255, nothing to commit
assert not conn._in_transaction
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None

227
tests/test_type_framing.py Normal file
View File

@ -0,0 +1,227 @@
"""Regression tests for tuple-payload framing bugs found 2026-05-08.
All three bugs below shipped in released versions and survived a
251-test integration suite for one reason: no fixture used the affected
types. They were reported from the field (Informix 12 user) but reproduce
identically on Informix 15 none of them is version-specific.
Each bug is covered twice: once for the value itself, and once with
**trailing columns** after the affected column. The trailing-column case
is the important one two of these bugs desynced the row decoder, so
the damage showed up in *later* columns, not the one with the bad type.
A single-column test would have passed while the driver was corrupting
every real query.
Wire evidence for each is recorded in the source comments at the fix
sites (``converters._decode_int8``, ``_resultset._FIXED_WIDTH_CHAR_TYPES``,
and the BOOLEAN branch of ``_resultset._legacy_dispatch_one_column``).
"""
from __future__ import annotations
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=10.0,
)
# --------------------------------------------------------------------------
# INT8 / SERIAL8 — the legacy 64-bit integer. 10 bytes, sign-magnitude,
# with the high and low 32-bit halves stored in the opposite order you'd
# expect. Previously fell through to the unknown-type path and surfaced
# as raw bytes. Correct width, so no desync — just a wrong value.
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
"value",
[
123456789012, # needs both halves
-123456789012, # negative: same magnitude bytes, sign word 0xFFFF
42, # low half only
0,
-1,
2**63 - 1, # INT8 max
-(2**63 - 1),
None, # sign word 0x0000
],
)
def test_int8_round_trip(conn_params: ConnParams, value: int | None) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_int8 (v INT8)")
cur.execute("INSERT INTO t_int8 VALUES (?)", (value,))
cur.execute("SELECT v FROM t_int8")
assert cur.fetchone() == (value,)
def test_int8_does_not_desync_following_columns(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_int8_mix "
"(a INT, b INT8, c INT, d VARCHAR(10))"
)
cur.execute(
"INSERT INTO t_int8_mix VALUES (?, ?, ?, ?)",
(111111, 123456789012, 222222, "tail"),
)
cur.execute("SELECT a, b, c, d FROM t_int8_mix")
assert cur.fetchone() == (111111, 123456789012, 222222, "tail")
def test_serial8_decodes_as_int(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_ser8 (v SERIAL8, note VARCHAR(8))")
cur.execute("INSERT INTO t_ser8 VALUES (0, 'x')")
cur.execute("SELECT v, note FROM t_ser8")
row = cur.fetchone()
assert row == (1, "x"), f"SERIAL8 row decoded as {row!r}"
def test_int8_is_not_confused_with_bigint(conn_params: ConnParams) -> None:
"""INT8 (17) is 10 bytes sign-magnitude; BIGINT (52) is 8 bytes two's
complement. Same logical range, completely different wire format
decoding one as the other is silently wrong for negatives."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_i8_bi (a INT8, b BIGINT)")
cur.execute("INSERT INTO t_i8_bi VALUES (?, ?)", (-77, -77))
cur.execute("SELECT a, b FROM t_i8_bi")
assert cur.fetchone() == (-77, -77)
# --------------------------------------------------------------------------
# NCHAR — fixed-width and space-padded, exactly like CHAR. Was being read
# as 1-byte-length-prefixed, so the first character got consumed as a
# length and the offset jumped by that value. NCHAR(10) holding 'nch'
# read 0x6E ('n') as a 110-byte length: silent truncation at best, a
# struct.error crash when other columns followed.
# --------------------------------------------------------------------------
def test_nchar_keeps_first_character(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_nchar (v NCHAR(10))")
cur.execute("INSERT INTO t_nchar VALUES ('nch')")
cur.execute("SELECT v FROM t_nchar")
assert cur.fetchone() == ("nch",)
def test_nchar_does_not_desync_following_columns(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_nchar_mix "
"(a INT, n NCHAR(10), c INT, v NVARCHAR(20), z INT)"
)
cur.execute(
"INSERT INTO t_nchar_mix VALUES (111111, 'nch', 222222, 'nvc', 333333)"
)
cur.execute("SELECT a, n, c, v, z FROM t_nchar_mix")
assert cur.fetchone() == (111111, "nch", 222222, "nvc", 333333)
def test_nvarchar_still_length_prefixed(conn_params: ConnParams) -> None:
"""Guard the other side of the NCHAR fix: NVARCHAR *is* byte-length-
prefixed and must not be moved to the fixed-width branch."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_nvc (v NVARCHAR(40), tail INT)")
cur.execute("INSERT INTO t_nvc VALUES ('nvc', 4242)")
cur.execute("SELECT v, tail FROM t_nvc")
assert cur.fetchone() == ("nvc", 4242)
# --------------------------------------------------------------------------
# BOOLEAN — described as UDTFIXED(41) with encoded_length=1, but carries
# the full UDT envelope on the wire: [1-byte indicator][4-byte length]
# [data]. Six bytes for a one-byte value. Reading only encoded_length left
# 5 bytes behind and corrupted every subsequent column — this is the bug
# that produced the original field report.
# --------------------------------------------------------------------------
@pytest.mark.parametrize(
("literal", "expected"),
[("'t'", True), ("'f'", False), ("NULL", None)],
)
def test_boolean_decodes(
conn_params: ConnParams, literal: str, expected: bool | None
) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_bool (v BOOLEAN)")
cur.execute(f"INSERT INTO t_bool VALUES ({literal})")
cur.execute("SELECT v FROM t_bool")
assert cur.fetchone() == (expected,)
def test_boolean_does_not_desync_following_columns(
conn_params: ConnParams,
) -> None:
"""The original field-reported symptom: a BOOLEAN column silently
shifted every column after it by 5 bytes."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_bool_mix "
"(a INT, b BOOLEAN, c INT, d VARCHAR(10))"
)
cur.execute("INSERT INTO t_bool_mix VALUES (111111, 't', 222222, 'tail')")
cur.execute("SELECT a, b, c, d FROM t_bool_mix")
assert cur.fetchone() == (111111, True, 222222, "tail")
def test_multiple_booleans_in_one_row(conn_params: ConnParams) -> None:
"""Each BOOLEAN consumes its own 6-byte envelope; an off-by-N in the
envelope compounds across columns."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_bool_many "
"(a BOOLEAN, b BOOLEAN, c BOOLEAN, tail INT)"
)
cur.execute("INSERT INTO t_bool_many VALUES ('t', 'f', 't', 5150)")
cur.execute("SELECT a, b, c, tail FROM t_bool_many")
assert cur.fetchone() == (True, False, True, 5150)
# --------------------------------------------------------------------------
# Combined: everything that previously mis-framed, in one row.
# --------------------------------------------------------------------------
def test_all_previously_broken_types_in_one_row(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_framing ("
" a INT, b BOOLEAN, c NCHAR(10), d INT8,"
" e VARCHAR(20), f NVARCHAR(20), g SERIAL8, h INT)"
)
cur.execute(
"INSERT INTO t_framing VALUES "
"(111111, 't', 'nch', 123456789012, 'vc', 'nvc', 0, 999888)"
)
cur.execute("SELECT a, b, c, d, e, f, g, h FROM t_framing")
assert cur.fetchone() == (
111111, True, "nch", 123456789012, "vc", "nvc", 1, 999888,
)

342
tests/test_type_matrix.py Normal file
View File

@ -0,0 +1,342 @@
"""Type-matrix regression tests, and the fuzzer that found them.
Six framing bugs reached users before this file existed. Every one shared
a shape: the value under test decoded fine on its own, and quietly
corrupted whatever came *after* it. They were missed not because branches
went unexercised but because the fixtures never supplied data that took
them the LVARCHAR fixture was ``'lv value'``, eight characters, even,
never NULL, so neither of its two broken branches ever ran.
Three principles follow, and every test here applies them:
1. **Always put a column after the value under test.** A trailing column
can be mis-sized with no visible effect. The sentinel is the detector.
2. **Vary the data, not just the type.** Odd vs even length, empty vs
NULL, min/max, negative, unscaled vs scaled.
3. **Use an oracle our codecs don't share.** A Python round-trip cannot
catch a symmetric encode/decode bug our DATETIME encoder wrote zeros
and our decoder read them back in perfect agreement. Asking the server
to render the stored value as text breaks that symmetry.
The three bugs in the first section were found by the fuzzer at the
bottom of this file, not by a person.
"""
from __future__ import annotations
import datetime
import decimal
import random
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
SENTINEL = 424242
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
# Bounded so a wire desync fails the test instead of hanging it.
read_timeout=20.0,
)
# ---------------------------------------------------------------------------
# 1. BOOLEAN could not be used as a bind parameter at all
# ---------------------------------------------------------------------------
# _encode_bool emitted type code 45, which the server does not accept as a
# bind type: it simply stopped responding. Any execute() with a bool
# parameter hung until the read timeout, or forever without one. Type 41
# (how BOOLEAN is *described* in results) hangs identically — descriptor
# type and bind type are not interchangeable. Binding 't'/'f' as CHAR and
# letting the server cast is what works.
@pytest.mark.parametrize("value", [True, False, None])
def test_boolean_bind_round_trip(
conn_params: ConnParams, value: bool | None
) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_bool_bind (b BOOLEAN, s INT)")
cur.execute("INSERT INTO t_bool_bind VALUES (?, ?)", (value, SENTINEL))
cur.execute("SELECT b, s FROM t_bool_bind")
assert cur.fetchone() == (value, SENTINEL)
def test_boolean_bind_in_where_clause(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_bool_where (b BOOLEAN, k INT)")
cur.execute("INSERT INTO t_bool_where VALUES (?, ?)", (True, 1))
cur.execute("INSERT INTO t_bool_where VALUES (?, ?)", (False, 2))
cur.execute("SELECT k FROM t_bool_where WHERE b = ?", (True,))
assert cur.fetchall() == [(1,)]
cur.execute("SELECT k FROM t_bool_where WHERE b = ?", (False,))
assert cur.fetchall() == [(2,)]
def test_boolean_bind_stores_real_boolean(conn_params: ConnParams) -> None:
"""Independent oracle: the server must agree it stored a BOOLEAN, not
the string we transported it as."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_bool_oracle (b BOOLEAN)")
cur.execute("INSERT INTO t_bool_oracle VALUES (?)", (True,))
cur.execute("SELECT b::CHAR(1) FROM t_bool_oracle")
assert cur.fetchone() == ("t",)
# ---------------------------------------------------------------------------
# 2. Unscaled DECIMAL was read one byte short
# ---------------------------------------------------------------------------
# Width is ((precision) + (scale & 1) + 3) // 2, per IfxColumnInfo's
# adjustedColumnLength. We omitted the `scale & 1` term, which only
# matters when precision is even and scale is odd — and an unscaled
# DECIMAL(p) reports scale 255. DECIMAL(16) is 10 bytes, not 9.
@pytest.mark.parametrize(
("ddl", "value"),
[
("DECIMAL(16)", decimal.Decimal("1234567890123456")),
("DECIMAL(16)", decimal.Decimal("-1")),
("DECIMAL(16)", None),
("DECIMAL(4)", decimal.Decimal("1234")),
("DECIMAL(20)", decimal.Decimal("12345678901234567890")),
("DECIMAL(8,2)", decimal.Decimal("12345.67")),
("DECIMAL(10,4)", decimal.Decimal("123456.7890")),
("DECIMAL(5,0)", decimal.Decimal("12345")),
("DECIMAL(1,0)", decimal.Decimal("7")),
("DECIMAL(32,10)", decimal.Decimal("1234567890.0123456789")),
("MONEY(10,2)", decimal.Decimal("1234.56")),
("MONEY(10,2)", decimal.Decimal("-1234.56")),
],
)
def test_decimal_width_does_not_shift_next_column(
conn_params: ConnParams, ddl: str, value: decimal.Decimal | None
) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(f"CREATE TEMP TABLE t_dec_w (d {ddl}, s INT)")
cur.execute("INSERT INTO t_dec_w VALUES (?, ?)", (value, SENTINEL))
cur.execute("SELECT d, s FROM t_dec_w")
got = cur.fetchone()
assert got[1] == SENTINEL, f"{ddl} mis-sized; sentinel came back {got[1]}"
if value is None:
assert got[0] is None
else:
assert got[0] == value
def test_packed_width_matches_the_reference_formula() -> None:
"""Unit-level guard on the formula itself, including the term that was
missing. Values verified against the wire."""
from informix_db._resultset import _packed_width
assert _packed_width((8 << 8) | 2) == 5 # DECIMAL(8,2)
assert _packed_width((16 << 8) | 255) == 10 # DECIMAL(16), unscaled
assert _packed_width((10 << 8) | 4) == 6 # DECIMAL(10,4)
assert _packed_width((5 << 8) | 0) == 4 # DECIMAL(5,0)
assert _packed_width((32 << 8) | 10) == 17 # DECIMAL(32,10)
assert _packed_width((1 << 8) | 0) == 2 # DECIMAL(1,0)
assert _packed_width((14 << 8) | 10) == 8 # DATETIME YEAR TO SECOND
assert _packed_width((19 << 8) | 15) == 11 # DATETIME .. FRACTION(5)
# ---------------------------------------------------------------------------
# 3. NULL CHAR / NCHAR came back as an empty string
# ---------------------------------------------------------------------------
# The NULL marker is a leading 0x00; an empty CHAR is all spaces. They are
# distinguishable on the wire, so conflating them made `IS NULL` disagree
# with what the driver returned.
@pytest.mark.parametrize("ddl", ["CHAR(6)", "NCHAR(6)"])
def test_null_char_is_none_not_empty_string(
conn_params: ConnParams, ddl: str
) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(f"CREATE TEMP TABLE t_char_null (k INT, c {ddl})")
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (1, None))
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (2, ""))
cur.execute("INSERT INTO t_char_null VALUES (?, ?)", (3, "ab"))
cur.execute("SELECT k, c FROM t_char_null ORDER BY k")
assert cur.fetchall() == [(1, None), (2, ""), (3, "ab")]
@pytest.mark.parametrize("ddl", ["CHAR(6)", "NCHAR(6)"])
def test_null_char_agrees_with_is_null(
conn_params: ConnParams, ddl: str
) -> None:
"""The server's own view is the oracle: what it calls NULL, we must
return as None."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(f"CREATE TEMP TABLE t_char_isnull (k INT, c {ddl})")
cur.execute("INSERT INTO t_char_isnull VALUES (?, ?)", (1, None))
cur.execute("INSERT INTO t_char_isnull VALUES (?, ?)", (2, ""))
cur.execute("SELECT k FROM t_char_isnull WHERE c IS NULL")
server_nulls = {r[0] for r in cur.fetchall()}
cur.execute("SELECT k, c FROM t_char_isnull ORDER BY k")
driver_nulls = {k for k, c in cur.fetchall() if c is None}
assert driver_nulls == server_nulls
# ---------------------------------------------------------------------------
# 4. The fuzzer itself, with fixed seeds so it is reproducible in CI
# ---------------------------------------------------------------------------
_ODD = "abcdefghijk" # 11
_EVEN = "abcdefghij" # 10
# (name, DDL, edge values). Values are chosen to flip framing branches.
_CORPUS: list[tuple[str, str, list]] = [
("smallint", "SMALLINT", [0, 1, -1, 32767, -32767, None]),
("int", "INT", [0, -1, 2147483647, None]),
("int8", "INT8", [0, 1, -1, 123456789012, -123456789012, None]),
("bigint", "BIGINT", [0, -1, 2**62, None]),
("float", "FLOAT", [0.0, 3.141592653589793, -1.5, None]),
("smallfloat", "SMALLFLOAT", [0.0, 2.5, None]),
("dec_scaled", "DECIMAL(8,2)", [
decimal.Decimal("12345.67"), decimal.Decimal("-98.76"), None,
]),
("dec_float", "DECIMAL(16)", [
decimal.Decimal("1234567890123456"), decimal.Decimal("-1"), None,
]),
("money", "MONEY(10,2)", [decimal.Decimal("1234.56"), None]),
("char", "CHAR(12)", ["", "a", _ODD, _EVEN, None]),
("varchar", "VARCHAR(64)", ["", "a", _ODD, _EVEN, None]),
("nchar", "NCHAR(12)", ["", "a", _ODD, None]),
("nvarchar", "NVARCHAR(64)", ["", "a", _ODD, None]),
("lvarchar", "LVARCHAR(1024)", [
"", "a", "ab", _ODD, _EVEN, "x" * 255, "y" * 256, None,
]),
("date", "DATE", [datetime.date(1899, 12, 31), datetime.date(2026, 8, 31), None]),
("dt_sec", "DATETIME YEAR TO SECOND", [
datetime.datetime(2026, 8, 31, 12, 30, 15), None,
]),
("dt_frac", "DATETIME YEAR TO FRACTION(5)", [
datetime.datetime(2026, 8, 31, 12, 30, 15),
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
None,
]),
("interval", "INTERVAL DAY(5) TO SECOND", [
datetime.timedelta(days=10, hours=4, minutes=30, seconds=15),
datetime.timedelta(0),
None,
]),
("boolean", "BOOLEAN", [True, False, None]),
]
_BY_NAME = {c[0]: c for c in _CORPUS}
def _check_row(cur, names: list[str], values: list) -> None:
"""Create, insert, then read back from several projections. The
sentinel column trails everything so any mis-sizing surfaces."""
cols = [f"c{i}" for i in range(len(names))]
ddl = ", ".join(f"{c} {_BY_NAME[n][1]}" for c, n in zip(cols, names, strict=True))
cur.execute(f"CREATE TEMP TABLE t_fuzz ({ddl}, sentinel INT)")
try:
marks = ", ".join(["?"] * (len(cols) + 1))
cur.execute(f"INSERT INTO t_fuzz VALUES ({marks})", (*values, SENTINEL))
expected = dict(zip(cols, values, strict=True))
expected["sentinel"] = SENTINEL
every = [*cols, "sentinel"]
cur.execute(f"SELECT {', '.join(every)} FROM t_fuzz")
assert dict(zip(every, cur.fetchone(), strict=True)) == expected, (
f"straight projection: {list(zip(names, values, strict=True))}"
)
for shift in range(1, min(len(every), 4)):
order = every[shift:] + every[:shift]
cur.execute(f"SELECT {', '.join(order)} FROM t_fuzz")
got = dict(zip(order, cur.fetchone(), strict=True))
assert got == expected, (
f"rotation {shift}: {list(zip(names, values, strict=True))}"
)
cur.execute("SELECT * FROM t_fuzz")
got_names = [d[0] for d in cur.description]
assert dict(zip(got_names, cur.fetchone(), strict=True)) == expected, (
f"SELECT *: {list(zip(names, values, strict=True))}"
)
finally:
cur.execute("DROP TABLE t_fuzz")
@pytest.mark.parametrize("type_name", [c[0] for c in _CORPUS])
def test_every_edge_value_with_a_trailing_sentinel(
conn_params: ConnParams, type_name: str
) -> None:
"""Exhaustive over the corpus: each type, each edge value, always with
a column behind it."""
with _connect(conn_params) as conn:
cur = conn.cursor()
for value in _BY_NAME[type_name][2]:
_check_row(cur, [type_name], [value])
@pytest.mark.parametrize("seed", [1, 2, 3])
def test_random_mixed_rows(conn_params: ConnParams, seed: int) -> None:
"""Random multi-type rows. Seeded so a failure is reproducible; the
point is combinations no one thought to write by hand."""
rng = random.Random(seed)
with _connect(conn_params) as conn:
cur = conn.cursor()
for _ in range(25):
names = [rng.choice(_CORPUS)[0] for _ in range(5)]
values = [rng.choice(_BY_NAME[n][2]) for n in names]
_check_row(cur, names, values)
def test_server_agrees_with_what_we_bound(conn_params: ConnParams) -> None:
"""The independent oracle. Compare against the server's own rendering
rather than our own decoder, so a symmetric encode/decode bug cannot
hide which is exactly how DATETIME lost its fractions."""
checks: list[tuple[str, object, str]] = [
("INT8", 123456789012, "123456789012"),
("INT8", -123456789012, "-123456789012"),
("BIGINT", 2**62, str(2**62)),
("VARCHAR(64)", "PackageRoot", "PackageRoot"),
("LVARCHAR(64)", "PackageRoot", "PackageRoot"),
("NVARCHAR(64)", "odd", "odd"),
("BOOLEAN", True, "t"),
("BOOLEAN", False, "f"),
("DATETIME YEAR TO FRACTION(5)",
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
"2026-08-31 12:30:15.12000"),
("DATETIME YEAR TO SECOND",
datetime.datetime(2026, 8, 31, 12, 30, 15),
"2026-08-31 12:30:15"),
("DATE", datetime.date(2026, 8, 31), "2026-08-31"),
]
with _connect(conn_params) as conn:
cur = conn.cursor()
for ddl, value, server_repr in checks:
cur.execute(f"CREATE TEMP TABLE t_oracle (v {ddl})")
try:
cur.execute("INSERT INTO t_oracle VALUES (?)", (value,))
cur.execute("SELECT v::LVARCHAR FROM t_oracle")
(text,) = cur.fetchone()
assert text.strip() == server_repr, (
f"{ddl}: bound {value!r}, server stored {text.strip()!r}, "
f"expected {server_repr!r}"
)
finally:
cur.execute("DROP TABLE t_oracle")

4
uv.lock generated
View File

@ -33,8 +33,8 @@ wheels = [
] ]
[[package]] [[package]]
name = "informix-db" name = "informix-driver"
version = "2026.5.5.9" version = "2026.9.3"
source = { editable = "." } source = { editable = "." }
[package.optional-dependencies] [package.optional-dependencies]