Compare commits

...

22 Commits

Author SHA1 Message Date
c85aaf2ecb Docs: put named row access where someone leaving IfxPy will find it
The API reference gained the Row section with the release, which is the
right home for the details and the wrong place to discover the feature
exists. Three gaps closed.

The IfxPy migration guide is the page for people in exactly the position
that prompted this: IfxPy is positional-only, so those codebases grow a
description/zip helper that yields a dict and still no attribute access.
That page now shows the helper it replaces alongside the one-liner, and
notes the .lower() in the hand-written version is already a no-op since
Informix folds unquoted identifiers.

Quickstart gains one sentence, next to the paramstyle note where someone
is already thinking about how rows come back.

The phase log records 2026.09.03, including where the testing actually
found the shadowed-names bug: not in the differential test across twenty
awkward types, which found nothing, but in six lines of name bookkeeping
that were correct when written and stopped being correct an hour later.

Verified live by content, and the #rows anchor the new cross-links point
at resolves.
2026-09-03 16:06:34 -06:00
acc21b8b81 Docs: row_factory reference, and the write-path corruption warning
The API page documents Row: the three access styles, that it is set on
the connection, that it subclasses tuple so existing code is unaffected,
the measured 9% and why that makes it opt-in, and the four behaviours
worth knowing (lower-case folding, expression column names, duplicate
names, and a column beating a method of the same name).

The types page gains the placeholder-rewriter entry. It is the only one
in that list affecting the write path rather than the read path, so
anyone who has stored HH:MM times or URLs with ports through this driver
should check what actually landed.

Test counts updated to 472/471. Verified live by content.
2026-09-03 16:00:53 -06:00
9fbb97199f Release 2026.09.03 — placeholder rewriter, and named row access
The rewriter that converts :N placeholders to ? was a bare regular
expression, so it rewrote the inside of string literals. Any HH:MM time,
any URL with a port, any key:value string was silently stored wrong. It
was not opt-in either: the rewrite runs whenever a statement has
parameters, whatever placeholder style the caller used. Replaced with a
scanner that substitutes only outside quotes and comments, using
Informix's own lexical rules measured against all three servers.

Row adds column-name and attribute access alongside the existing
positional access, on request from a user coming from pyodbc and
mssql-python. Opt-in at connect time, because it costs about 9% on bulk
fetch and that is not a trade to make for someone who never reads a
column by name.

472 tests on 15 and 14.10, 471 on 12.10.
2026-09-03 15:59:26 -06:00
ce5a582048 Testing Row found the same mistake I keep making
I hand-listed the names a column could shadow: count and index, the two
tuple methods. Then I wrote keys(), _asdict() and _fields on the same
class and did not go back. A column called "keys" returned a bound
method instead of a value, silently, which is precisely the failure
shape this driver spent a fortnight removing from its decoders.

The fix is not a longer list. The reserved set is now computed from
dir(Row), so adding a method later cannot reopen the hole, and the test
is parametrized over that computed set so it grows with the class.

_fields and _map moved to name-mangled attributes. They are read by
repr() and _asdict(), so shadowing the public _fields with a column
would have made the machinery report the column value instead of the
column names. Mangling keeps the two apart, and there is a test that
shadows _fields and checks repr still works.

Also exercised, and all clean: value-identical output against the plain
tuple path across twenty columns covering every awkward type plus a
fully NULL row, which is the strongest available statement that Row is a
presentation layer and not a bug; zero-column and 500-column rows;
unicode, spaced, digit-leading, empty-string and dunder column names;
rows outliving eviction of their class from the bounded cache; the four
async fetch routes and the async pool; and twelve threads racing on the
class cache, which correctly share one class per shape.

472 tests on 15 and 14.10, 471 on 12.10.
2026-09-03 15:51:46 -06:00
805fa58eb2 Rows can answer to a column name, if you ask for it
Field request from a user running this alongside SQL Server, where both
pyodbc and mssql-python hand back rows that take a position, a column
name, and an attribute. The argument is readability on wide
projections: row[11] tells a reader nothing, and stays correct only
until somebody adds a column in the middle.

row_factory=Row gives all three. It is set on the connection, so it is
one line for the whole application rather than per query, which is what
"without any additional coding" has to mean in practice. A cursor can
override it.

Opt-in, not default, and the number is why. Measured on a 20,000-row
five-column fetch: 37.2 ms with tuples, 40.8 ms with Row, about 9%.
Defaulting it on would move the published 1.05-1.15x ratio against IfxPy
to roughly 1.15-1.25x. That is not a trade to make on behalf of somebody
running a bulk export that never looks at a column by name.

Row subclasses tuple, so row == (1, "x") still holds. That constraint
shaped the design more than anything else: this suite compares fetched
rows against plain tuples in hundreds of places, and so does everybody's
code. Slices degrade to plain tuples, since a slice has no column map.

Details that had to be decided rather than assumed, all measured against
the servers: Informix folds unquoted identifiers to lower case, so the
.lower() in the workaround people write by hand is a no-op. Expression
columns get names like "(count(*))" that cannot be attributes, so they
are subscript-only. Duplicate names resolve to the first occurrence,
matching pyodbc. And tuple already defines count and index, so a column
with either name gets a descriptor and wins, because on a database row
that is plainly what the caller meant.

The per-shape class is cached, so a type() call does not land on every
small query, and rows pickle by rebuilding from their field names.
2026-09-03 15:42:56 -06:00
088171325d The placeholder rewriter could not see a string literal
We advertise paramstyle="numeric" and the wire protocol takes ?, so :N
placeholders are rewritten on the way out. That was a bare
re.sub(r":(\d+)", "?", sql), which rewrote the inside of string literals:

    UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
                      stored as 'http://host?/x'

Any HH:MM time, any URL with a port, any aspect ratio, any key:value
string. It wrote wrong data and said nothing about it.

None of it was opt-in. The rewrite fires whenever a statement has
parameters, whatever placeholder style the caller actually used, so
writing ? everywhere and never touching numeric style did not protect
you. It also changed the placeholder count while num_qmarks was still
computed from len(params), leaving driver and server disagreeing about
how many binds exist.

Replaced with a single pass that substitutes only outside quotes and
comments. The lexical rules are Informix's own, measured against all
three servers rather than assumed from standard SQL, and one of them
would have been got wrong by habit: a backslash escapes nothing, so
'a\'b' is an unterminated string and draws -282. A scanner written to
Postgres reflexes would desync there and corrupt everything after it.
Block comments do not nest, the first */ closes them. Braces are a
comment. :: is stepped over as a unit so a cast can never be read as the
start of a placeholder.

An unterminated quote or comment consumes the rest and substitutes
nothing further. Under-substituting leaves the server to reject SQL that
was already malformed; guessing would corrupt a literal.

This was the last place the driver inferred meaning from SQL text
instead of handling it properly. Eleven of the new tests fail against
the previous commit.
2026-09-03 12:45:58 -06:00
7ffc148112 Docs: cover the 2026.09.02 review, and drop every em-dash
Content. The types page now warns about the two framing bugs this
release fixed, since both silently corrupt data on versions before it:
a BLOB or CLOB in any position but last shifted every column after it,
and a NULL collection did the same. The API page documents two things
that are now user-visible: only one scrollable cursor may be open per
connection (previously the second statement drew -285 and destroyed the
cursor as collateral), and transaction control written as SQL is tracked
(previously rollback() after a SQL BEGIN WORK sent nothing and reported
success). The phase log gains a section on the review itself, including
the pattern behind almost all eleven bugs. Test counts updated to
457/456.

Style. 136 em-dashes across 22 content files, plus Hero.astro and two
stylesheets, rewritten rather than substituted -- swapping the character
for a comma leaves prose that reads like it lost an argument with a
linter. Bullets that used a dash to gloss a term now use a colon;
parenthetical asides became their own sentences or moved inside
brackets. The rendered HTML is clean.

Verified against the live site by content, not status code: these pages
return 200 for every path and render the 404 body, so a stale deploy
looks perfectly healthy.
2026-09-02 10:36:44 -06:00
8f341c4b2a Release 2026.09.02 — a systematic review, and eleven bugs it found
No new features. The result of going back over the driver looking for
the shape of past bugs rather than for new symptoms.

Two returned wrong data silently: smart LOBs were read 77 bytes short,
shifting every column after them, and NULL composite UDTs skipped their
length field. One did nothing at all: rollback() after a SQL BEGIN WORK
returned successfully having sent nothing, leaving the rows it was asked
to discard in place and the connection back in the pool holding an open
transaction.

The rest: five ordinary query forms failing with -260 because
classification guessed from the first word; a scrollable cursor taking
the whole connection down with it; two unguarded exits leaking failed
statements; deferred cleanup landing inside somebody else's statement,
and a GC-time finalizer that couldn't tell it was already on the thread
holding the wire lock; the async layer borrowing the process-wide thread
pool; and two readers sharing one socket stream without agreeing.

457 integration tests on 15 and 14.10, 456 on 12.10 (one skip, no CTEs
before 14.10) -- up from 414. Every fix has at least one test that fails
against 2026.09.01.
2026-09-02 10:28:27 -06:00
429a3910e4 The async layer borrowed a thread pool belonging to the whole process
Every blocking call in informix_db.aio went through asyncio.to_thread,
which runs on the event loop's default executor. That executor belongs
to the process, not to us, and it is sized from the CPU count --
min(32, cpu_count + 4), so six threads on a two-CPU container.

Cancelled calls hold their threads. asyncio.to_thread cannot interrupt a
worker, so a cancelled await leaves the thread running the wire call
until the read timeout. Cancellation is ordinary in a web app -- a
client disconnect cancels the request task -- so a handful of them pins
every thread in the shared pool. Measured: six cancelled calls against a
six-worker default executor starve an unrelated to_thread indefinitely,
and with the executor held, four concurrent driver queries never ran at
all.

Pool concurrency was capped by the same number without saying so. A pool
with max_size=20 on a two-CPU box ran six queries at a time.

Each connection now owns one thread. That is the right size rather than
a compromise -- the sync connection serializes every wire operation on
its own lock, so a second thread could do nothing but wait for the
first. It also avoids a deadlock a shared pool-sized executor invites:
with N threads and N connections, N tasks blocked in acquire occupy
every thread while the connection they are waiting for is held by a task
that now needs a thread to finish and release it.

The executor lives on the sync connection so it survives being returned
to the pool and handed out again, rather than being rebuilt per acquire.
release() runs on the connection's own thread, which is idle by
definition and keeps release off any pool that waiters may have filled
-- release has to win that race, since it frees what they are waiting
for. connect() and pool acquire stay on the default executor: there is
no connection yet to own a thread, and neither can deadlock against
query threads any more.

close() shuts the executor down, and a weakref finalizer is the backstop
for a connection dropped without it -- ThreadPoolExecutor workers park
on the work queue rather than exiting when idle, so an executor that is
never shut down leaks its thread for the life of the process. The
thread-count test caught that gap; it was missing from the first cut.
2026-09-02 01:03:12 -06:00
5ad296419c Two readers shared a stream without agreeing, and a length was taken on trust
IfxSocket owns a read-ahead buffer that BufferedSocketReader fills and
drains. Connection._drain_to_eot, _raise_sq_err and the login path
bypass that reader and call IfxSocket.read_exact directly, which recv'd
from the socket without looking at the buffer. Bytes sitting in the
buffer were skipped, and skipped bytes in a length-framed protocol do
not announce themselves -- the next read lands mid-field and every read
after it is wrong.

Nothing triggers it today. The server sends one response per request, so
recv returns exactly that response and the buffered reader consumes all
of it before control returns to a direct read. That is a property of the
traffic, not of the code, and the buffer is connection-scoped precisely
so read-ahead can cross response boundaries -- pipelined executemany
already puts several responses in flight. read_exact now drains the
buffer first, which costs one branch on a cold path and makes the two
paths agree by construction rather than by luck.

fill_recv_buf believed whatever byte count it was handed, and that count
is almost always a length field straight off the wire. A garbage
0x7FFFFFFF reads as a 2 GB request and the fill loop sits in recv until
the read timeout while the buffer grows. It now refuses above
IFX_MAX_READ_BYTES (256 MiB default) with an error naming the number,
which is the actual diagnostic: a length that absurd means framing was
already lost upstream.

BufferedSocketReader.skip advanced the cursor arithmetically with no
guard, so a negative count rewound it and re-decoded consumed bytes as
the next field. The base reader's skip delegates to read_exact and does
guard; this one diverged.

Transaction control run as SQL desynced Connection._in_transaction,
which is what commit() and rollback() are guarded by and what the pool
reads to decide whether a returned connection needs cleaning up. With
autocommit on, cursor.execute("BEGIN WORK") opened a real transaction
while the flag stayed False, so rollback() returned successfully having
sent nothing and the rows it was asked to discard survived. The
connection then went back to the pool holding an open transaction and
its locks. With autocommit off it failed instead: the driver's implicit
SQ_BEGIN fired first and the caller's BEGIN WORK got -535.

The server labels these -- 34 BEGIN, 35 COMMIT, 36 ROLLBACK, with and
without the WORK keyword, measured on all three versions. JDBC reads the
same values off the describe and calls setTxBeginState/setTxEndState.
_ensure_transaction moves to after the describe, matching JDBC's
initiateTransaction placement, which is what makes it possible to skip
for transaction control at all -- until the describe lands you cannot
know that is what the statement is.
2026-09-02 00:54:12 -06:00
c67bbc4766 Statement classification guessed from the first word and got five forms wrong
The driver chose between "open a cursor and fetch" and "execute and
release" by checking whether the first word of the SQL was SELECT. That
misses a leading comment in any of the three Informix flavours (--,
/* */, { }), a parenthesized select, a parenthesized UNION, and a CTE.
All five are perfectly ordinary queries. All five failed with -260,
"Cursor name already in use" -- an error that describes neither the
cause nor anything the caller did. It says "cursor" because the driver
sent SQ_EXECUTE where the server was waiting to open one.

The server had been answering this question all along. statement_type is
the first field of the DESCRIBE response and parse_describe has always
put it in the metadata dict, where nothing read it.

The comment defending the heuristic said nfields couldn't distinguish
these cases because INSERT INTO t VALUES (?) also describes a column.
That was true, and it argued for the wrong conclusion: statement_type
distinguishes them exactly. Measured against all three servers -- every
SELECT form above reports 2, that INSERT reports 6, UPDATE 33, DELETE
32, CREATE 45, EXECUTE PROCEDURE 56.

The predicate is now JDBC's IfxSqli.isResultSet: type 2, or type 56 with
at least one column, since a routine may or may not return rows. That
last clause fixes EXECUTE FUNCTION as a side effect -- it used to run
down the DML path and discard the return value. It now yields it.

executemany keeps the first-word check as a cheap pre-flight reject, and
gains an authoritative one after PREPARE for the forms the first word
misses. The refusal releases the statement, so the connection survives.

Informix 12.10 has no CTEs and rejects WITH with -201 at offset 1. That
is the server declining the grammar rather than the driver routing it
wrongly, and the test skips on it.
2026-09-02 00:38:31 -06:00
2eb5ac8f0f The finalizer's lock probe couldn't see its own thread
The cursor finalizer runs at GC time on whatever thread happened to
allocate, so it must not touch the wire while a statement owns it. It
checked with _wire_lock.acquire(blocking=False), meaning to ask "is
anyone using the wire?" -- but an RLock grants a reentrant acquire to
its own owner. When GC fired on the thread that was mid-statement, the
probe returned True, the finalizer concluded it had exclusive access,
and it sent CLOSE/RELEASE into the middle of the statement it had just
interrupted. The victim got -208 on valid SQL.

Refcounting hid it. A dropped cursor is freed at the drop, before the
next statement runs. A cursor caught in a reference cycle waits for a
collection instead, and cycles are ordinary -- any traceback that holds
a cursor makes one. Reproduced by putting an abandoned scrollable cursor
in a cycle and collecting mid-statement.

_wire_lock is now a small wrapper that tracks owner and depth, so
held_by_current_thread answers the question the finalizer was actually
asking. Same-thread GC defers to the cleanup queue exactly as another
thread's would. This also retires the _is_owned() call in
_ensure_transaction, which was reaching into CPython private API for
the same information.

The finalizer's error handling had the stale-cleanup defect too: a
leftover CLOSE draws -267, an OperationalError, which is in WIRE_ERRORS,
so it force-closed a healthy connection over a no-op. Server-reported
errors are told apart from wire failures by their sqlcode, matching
_drain_pending_cleanup.
2026-09-02 00:29:04 -06:00
ac00a25cb7 A session has one statement slot and the driver acted as if it had many
SQLI gives a session a single statement slot. SQ_CLOSE, SQ_RELEASE and
SQ_SFETCH act on whatever statement is current -- none of them names
one. Ordinary use never notices, because a non-scrollable cursor
materializes its rows and releases the statement before returning.

A scrollable cursor holds the slot open on purpose, and two things went
wrong there without saying so.

Another statement on the same connection got -285, and the scrollable
cursor was collateral damage: its next fetch came back -267, "the
transaction has been rolled back, all locks released". Two errors,
neither naming the cause, from code that reads as completely ordinary --
iterate a large result set, run a lookup partway through. It is now a
ProgrammingError that explains the constraint, and the scrollable cursor
is left alone. Re-executing the same scrollable cursor is the one caller
allowed past the check, and it now closes its own server-side cursor
first, because it collided with itself too.

The deferred-cleanup queue was flushed before every PDU. But a finalizer
enqueues precisely because it lost the wire lock, meaning another thread
is mid-statement -- so the flush landed inside that thread's own
statement and released it. The victim saw -208 when it landed before the
first fetch, -267 between fetch batches. Flushing only at a statement
boundary is both correct and sufficient: cleanup that misses one
statement is picked up by the next.

A stale queue entry was fatal on top of that. The finalizer enqueues,
the cursor is then closed properly, and the leftover CLOSE draws -267 --
an OperationalError, which is in WIRE_ERRORS, which force-closed a
healthy connection. Server-reported errors are now told apart from wire
failures by whether they carry a sqlcode.

Noted while tracing this: _build_close_pdu and _build_release_pdu write
SQ_ID followed by write_int(opcode), which frames correctly only because
the four bytes happen to land as [statement_id=0][opcode]. JDBC's
sendStatementID writes the real id there, and _read_describe_response
has been parsing it into _statement_id all along without ever using it.
Addressing CLOSE/RELEASE correctly is not enough for multiplexing on its
own -- a second cursor's SFETCH addressed to its own id returns -259 --
so that stays open rather than half-done.
2026-09-02 00:24:09 -06:00
0d8bd57ba9 The last two doors out of a statement didn't release it
A statement that fails is still allocated server-side, and skipping the
RELEASE bricks the connection: the next PREPARE collides with the leaked
one and every later call returns a nonsense error whose offset points
back at the failed SQL. That guard has now been added six times, once
per exit, each after a user found the exit. Two were still open.

The parameterized-SELECT bind drain. The build was guarded and the drain
was not, and the drain is where the interesting failures land: passing a
string where the column is an INT encodes cleanly, so the rejection
comes from the server (-1213, -415) after the guard has been passed. A
wrong-typed parameter is about as common as application mistakes get.

The scrollable-cursor open, which had no guard at all. Worst place to
lack one -- the GC-time finalizer is armed on the line after the drain,
so a failure there left the statement allocated with no fallback of any
kind. SELECT ... FOR UPDATE reaches it: prepares clean, fails at OPEN
with -526.

All six now call one helper instead of hand-rolling the cleanup, which
also fixes a defect in the two copies that had a cursor to close. CLOSE
and RELEASE shared a single contextlib.suppress block, so a CLOSE that
raised skipped the RELEASE -- losing the half that matters, since a
leaked cursor handle is a nuisance and a leaked statement breaks the
next call. Each PDU gets its own suppression now.

All four new integration tests fail against the previous commit.
2026-09-02 00:10:23 -06:00
66120cf6f3 Rows never checked that they consumed their own payload
Fourteen framing bugs shipped before this. Every one was a column
reading the wrong number of bytes, and not one of them raised at the
point of the mistake -- the wrong width produced a plausible value and
corrupted whatever came next, so the damage surfaced in a different
column, a different row, or a different statement entirely.

All fourteen were detectable for free. The payload is a fully extracted
bytes object of known length, so after decoding N columns the offset has
to land exactly on the end. It didn't, and nobody looked.

_row_not_consumed now raises when it doesn't, naming the column shape
and the byte delta, and it is wired into all three decode paths --
readers fast path, legacy chain, and the generated decoder.

Turning it on found two more bugs on the first run:

Smart LOBs were read as a flat 72-byte field. They use the UDT envelope:
149 bytes populated ([ind=0][len=144][144 hex chars]), 5 bytes NULL. We
consumed 72 and left 77 behind, so a BLOB or CLOB anywhere but the final
column position shifted every column after it. The 144 bytes are the
72-byte locator hex-encoded, which means BlobLocator.raw had never held
a locator, only the first half of the hex text -- unnoticed because
read_blob_column resolves through the server's lotofile and never reads
the locator.

NULL composite UDTs skipped their length field, byte for byte the
LVARCHAR NULL bug in a branch twelve lines away.

Both are the same [indicator][int32 length][data] envelope, hand-written
a third and fourth time. _read_udt_envelope is now the only copy, and it
rejects a negative length instead of rewinding the offset into bytes it
already decoded.
2026-09-02 00:04:26 -06:00
2e30ceacfb Regenerate uv.lock for 2026.09.01 2026-09-01 22:32:33 -06:00
2896f7e93e TLS traffic fuzzed; no bugs found (2026.09.01)
Tests and docs only, no behaviour change. TLS was the last untested
surface and it came back clean.

Previous coverage stopped at the handshake; everything after it ran only
over plain sockets. That gap mattered because SSLSocket.recv is not
socket.recv: it returns at most one TLS record's worth of plaintext
however much you ask for, can return fewer bytes than are available, and
plaintext buffered inside the SSL object is invisible to the OS. The
Phase 39 buffered reader asks for up to 64 KB per call and loops until
satisfied — that loop had never been pressured the same way.

tests/test_tls_traffic.py runs real SQLI through a TLS-terminating
proxy: the framing-bug types end to end, payloads at 1/4096/16383/16384/
16385/32000 bytes (straddling the ~16 KB record boundary), 500- and
5000-row bulk fetches, error recovery, concurrent TLS sessions, and
three negative cases — TLS client against a plaintext port, plaintext
client against a TLS port (must raise, not hang), and a verifying
context rejecting a self-signed cert.

All clean on all three servers. The buffered reader handles SSLSocket
semantics correctly.

Scope, stated plainly: the proxy supplies the TLS half, so this covers
the driver's TLS path — the half we own. It does NOT cover IBM's
server-side TLS listener. Setting one up on the developer-edition image
was attempted and abandoned: Informix 15 wants a PKCS#12 keystore
(onkstash takes a .p12, not the older CMS .kdb) and the engine kept
rejecting the stash with GSK_ERROR_BAD_KEYFILE_PASSWORD even with a
keystore GSKit itself could open. That half is IBM's code; everything
below ssl.wrap_socket is identical either way.

414/414 integration on 15, 14.10 and 12.10 (was 399).

Every surface is now fuzzed: type framing, fetch batching, error
recovery, cursor lifecycle, threads, pooling, transactions, async
cancellation, executemany partial failure, scrollable cursors, smart
LOBs, and TLS.
2026-09-01 22:32:10 -06:00
488773f078 Regenerate uv.lock for 2026.08.31.3 2026-08-31 19:13:32 -06:00
ac9075ca9a An encoding failure inside executemany leaked the statement (2026.08.31.3)
Last of the untested surface: pipelined executemany, scrollable cursors,
and smart LOBs. One bug, in the first.

executemany builds all its BIND+EXECUTE PDUs after the PREPARE and
before anything is drained — that batching is what makes the pipeline
fast. If encoding a row raises there (a value the connection's codec
cannot represent), the exception escaped without sending the RELEASE,
leaking the prepared statement. The next PREPARE collided with it and
every later call on the connection failed with an error pointing at the
previous SQL.

Same failure as 2026.08.31.2, in a sibling path.
_execute_dml_with_params has guarded this exact case for the single-row
path for a long time; the pipelined path was never given the same
treatment. That is the recurring shape of these bugs: a hazard
understood in one place and not carried to the code beside it.

What the fuzzer found sound, which is the larger part of the result:

  executemany constraint failures — duplicate-key and NOT NULL at
  first/mid/last of batches from 2 to 1000 rows all recover, and
  COUNT(*) always agrees with a full fetch, so the drain-N-responses
  invariant holds under partial failure.

  Scrollable cursors — fetch_first/last/prior/relative/absolute correct
  at 0, 1, 2, 5, 50, 300 rows, including off both ends (None, not a wrap
  or a crash) and a full forward walk after arbitrary positioning.
  Twenty abandoned scroll cursors leak nothing.

  Smart LOBs — round-trip at 0, 1, 255, 256, 1023, 1024, 4095, 4096,
  65535, 65536 bytes, straddling the 4096-byte SQ_FILE chunk and the
  64K mark, plus recovery from failed reads.

Still not fuzzed, stated plainly: TLS is handshake-tested against a
self-signed local socket, not a real Informix TLS listener (that needs
server-side keystore + onconfig SSL setup absent from the test
containers). The SQLI layer above the socket is identical either way.

399/399 integration on 15, 14.10 and 12.10 (was 356).
2026-08-31 19:11:05 -06:00
3d19ca0f1b Regenerate uv.lock for 2026.08.31.2 2026-08-31 17:17:30 -06:00
4af1c90c6f Failed statements bricked the connection; cancelled acquires starved the
pool (2026.08.31.2)

Two more found by fuzzing rather than by users, outside the type system.
Both are ordinary-path bugs a happy-path test cannot reach.

A failed statement was never released. Successful DML sent PREPARE ->
EXECUTE -> RELEASE; failing DML sent PREPARE -> EXECUTE and stopped. The
statement stayed allocated server-side, collided with the next PREPARE,
and every subsequent call on that connection returned a nonsense error
(-255 "Not in transaction" under autocommit, -285 otherwise) whose
offset pointed back at the FAILED sql rather than the new statement.

In practice: one duplicate-key violation bricked the connection. An
INSERT tripping a unique constraint is about the most routine error an
application can hit — every insert-if-not-exists pattern makes them —
and afterwards nothing on that connection worked.

The docstring on the parameterised path already described this hazard
and guarded the parameter-ENCODING failure; it never covered EXECUTE
itself failing. Fixed in all three paths (DML, parameterised DML, and
SELECT — whose fetch loop had the same gap, and whose GC-time finalizer
only covers scrollable cursors).

Cancelling a pool acquire leaked the connection permanently.
asyncio.to_thread cannot interrupt its worker, so a task cancelled while
blocked in pool.acquire() left the worker to finish and hand back a
connection nobody owned — checked out, never returned, one slot gone per
occurrence until the pool was dead.

Not theoretical for anything serving HTTP: a client disconnect cancels
the request task, and under load those cancellations land precisely
while waiting for a connection. The pool dies one slot at a time, only
under load, and PoolTimeoutError points nowhere near the cause.
acquire() now shields the inner future and returns whatever the worker
produced if the caller went away; add_done_callback fires immediately on
an already-resolved future, so the "worker finished just before the
cancel" race is the same code path rather than a second one.

The harness covered three things the type fuzzer cannot see: fetch
batching (NFETCH is a 4096-BYTE budget, so batch edges move with row
width — counts 0..1025 across three schema widths, every fetch style
required to agree), error recovery (eight error classes, repeated, each
followed by a known-good query on the same cursor and a fresh one), and
concurrency (threads sharing one connection, pooled threads that fail
before releasing, transaction isolation across checkouts, async
cancellation). Every worker reads back a value only it supplied, so a
crossed wire fails on data even when nothing raises.

Two harness false positives worth recording: Informix silently truncates
over-long strings (a literal SQL insert does the same, so the driver
matches the server), and it rejects a bare `?` in a projection — a cast
is required.

356/356 integration on 15, 14.10 and 12.10 (was 326).
2026-08-31 17:13:43 -06:00
4e186bb890 Regenerate uv.lock for 2026.08.31.1 2026-08-31 16:01:34 -06:00
49 changed files with 5139 additions and 308 deletions

View File

@ -2,6 +2,222 @@
All notable changes to `informix-db`. Versioning is [CalVer](https://calver.org/) — `YYYY.MM.DD` for date-based releases, `YYYY.MM.DD.N` for same-day post-releases per PEP 440.
## 2026.09.03 — A placeholder rewriter that could not see a string literal, and named row access
### Upgrade if you write string literals containing a colon
The driver rewrites `:1` placeholders to the `?` the wire protocol takes. That rewrite was a bare regular expression, and a regular expression cannot see a string literal, so it rewrote the inside of one:
```sql
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
-- stored as 'http://host?/x'
```
Any `HH:MM` time, any URL with a port, any aspect ratio, any `key:value` string. It wrote wrong data and said nothing about it.
Two things made it wider than it looks. It was never opt-in: the rewrite runs whenever a statement has parameters, whatever placeholder style you actually used, so writing `?` everywhere and never touching numeric style gave no protection. And it changed the placeholder *count* while the driver still told the server there were `len(params)` of them, leaving the two disagreeing about how many binds exist.
It is now a single pass that substitutes only outside quotes and comments. The lexical rules are Informix's own, measured against all three servers rather than assumed from standard SQL, and one of them would have been got wrong from habit: a backslash escapes nothing, so `'a\'b'` is an unterminated string and draws `-282`. A scanner written to Postgres reflexes would have desynced there and corrupted everything after it. Block comments do not nest, the first `*/` closes them. Braces are a comment. `::` is stepped over as a unit so a cast can never be read as the start of a placeholder.
An unterminated quote or comment consumes the rest and substitutes nothing further. Under-substituting hands malformed SQL to the server to reject; guessing would corrupt a literal.
This was the last place the driver inferred meaning from SQL text instead of handling it properly.
### Rows that answer to a column name
Requested by a user running this alongside SQL Server, where both `pyodbc` and `mssql-python` hand back rows addressable three ways at once. The argument is readability on a wide projection: `row[11]` tells a reader nothing, and stays correct only until somebody adds a column in the middle.
```python
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur.execute("SELECT tabid, tabname FROM systables")
row = cur.fetchone()
row[0], row["tabname"], row.tabname
```
Set on the connection, so it is one line for an application rather than per query. A cursor can override it. Pools and the async API forward it unchanged.
`Row` subclasses `tuple`, so `row == (1, "x")` is still true and code that treats rows as sequences keeps working. Slices degrade to plain tuples, since a slice has no column map.
**It is opt-in, and here is the number.** On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` is a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript. Defaulting it on would move the published 1.05-1.15x ratio against IfxPy to roughly 1.15-1.25x. That is a reasonable trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is a choice rather than a decision made on your behalf.
Details that were measured rather than assumed. Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`, and the `.lower()` in the workaround people write by hand is already a no-op. Expression columns get server-generated names like `(count(*))` that cannot be attributes, so they are subscript-only. Duplicate names resolve to the first occurrence, matching `pyodbc`. And a column always beats a method of the same name: `tuple` defines `count` and `index`, `Row` adds `keys`, `_asdict` and `_fields`, and any column with one of those names wins, because otherwise it would hand back a bound method in silence. That reserved set is computed from the class rather than hand-listed, which is what caught it: the hand-listed version covered the two `tuple` methods and missed all three of `Row`'s own.
### Verified
**472** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **471** on 12.10.FC12W1DE (one skip, no common table expressions before 14.10), up from 457. Both changes have tests that fail against `2026.09.02`.
## 2026.09.02 — A systematic review, and eleven bugs it found
No new features. This is the result of going back over the driver looking for the *shape* of past bugs rather than for new symptoms, and it turned up more than expected — including two that silently returned wrong data and one where `rollback()` did nothing at all.
### Rows never checked that they consumed their own payload
Fourteen framing bugs had reached users before this release. Every one was the same defect — a column reading the wrong number of bytes — and not one of them raised where the mistake happened. The wrong width produced a plausible value and corrupted whatever came *next*, so the damage always surfaced in a different column, a different row, or a different statement.
All fourteen were detectable for free. The payload is a fully extracted `bytes` of known length, so after decoding N columns the offset has to land exactly on the end. Now it's checked, on every row, in all three decode paths, and a mismatch names the column shape and the byte delta instead of handing back plausible garbage.
Turning it on found two more bugs on the first run:
**Smart LOBs were read as a flat 72-byte field.** They actually use the UDT envelope — 149 bytes when populated, 5 when NULL. The driver consumed 72 and left 77 behind, so a `BLOB` or `CLOB` anywhere but the final column position shifted every column after it. The 144 envelope bytes are the 72-byte locator *hex-encoded*, which means `BlobLocator.raw` had never held a locator, only the first half of the hex text. Nobody noticed because `read_blob_column` resolves LOBs server-side and never reads it.
**NULL composite UDTs skipped their length field**, byte for byte the LVARCHAR NULL bug in a branch twelve lines away. Both are now the one `_read_udt_envelope`, which also rejects a negative length rather than rewinding into bytes it already decoded.
### `rollback()` could silently do nothing
`cursor.execute("BEGIN WORK")` is a reasonable thing to write, and with autocommit on it opened a real transaction that the connection never learned about. `commit()` and `rollback()` are both guarded by that flag, so **`rollback()` returned successfully having sent nothing, and the rows it was asked to discard survived.** The connection then went back to the pool holding an open transaction and its locks, because the pool's cleanup is guarded by the same flag.
With autocommit off it failed instead, and for a sillier reason: the driver's implicit `SQ_BEGIN` fired first, so the caller's `BEGIN WORK` got `-535`, "already in transaction". The driver and the user competing to open the same transaction, and the user losing.
The server labels these statements and always has. Both spellings of all three now update the connection's state, and the implicit begin is skipped when the caller is doing the job themselves.
### Five ordinary query forms failed with a nonsense error
The driver decided whether to open a cursor by checking if the first word of the SQL was `SELECT`. That gets wrong:
```sql
-- a leading comment → -260
/* of any of the three flavours */ → -260
{ that Informix accepts } → -260
(SELECT parenthesized) → -260
WITH cte AS (...) SELECT ... → -260
(SELECT a) UNION (SELECT b) → -260
```
`-260` is "Cursor name already in use", which describes neither the cause nor anything the caller did. It says "cursor" because the driver sent `SQ_EXECUTE` where the server was waiting to open one.
The server reports the statement type in the first field of every DESCRIBE response, and the driver had been parsing it into a metadata dict that nothing read. The decision now uses it. `EXECUTE FUNCTION` gets fixed as a side effect — it used to run down the DML path and discard its return value, and now yields it.
### A scrollable cursor took the connection down with it
Informix gives a session one statement slot. A scrollable cursor holds it open on purpose, so any other statement on that connection got `-285`**and destroyed the scrollable cursor too**, whose next fetch came back `-267`, "the transaction has been rolled back, all locks released". Two unattributable failures from code that reads as entirely ordinary: iterate a large result set, run a lookup partway through.
It is now a `ProgrammingError` that explains the constraint and leaves the cursor alone. Re-executing the *same* scrollable cursor is still allowed and now closes its own server-side cursor first, because it collided with itself as well.
### Statements that failed were sometimes never released
A failed statement stays allocated server-side and collides with the next `PREPARE`, after which every call on that connection returns a nonsense error pointing at the *previous* SQL. That guard existed at six exits and was missing from two, both easy to reach:
- The parameterized-`SELECT` bind drain. Passing a string where the column is an `INT` encodes cleanly, so the rejection comes from the server (`-1213`, `-415`) *after* the guarded step.
- Opening a scrollable cursor — no guard at all, and the worst place to lack one, since the GC-time fallback is armed on the line after.
All six now share one implementation, which also fixed a defect in the copies that had a cursor to close: `CLOSE` and `RELEASE` shared a single suppression, so a failing `CLOSE` skipped the `RELEASE` — losing the half that actually matters.
### Cleanup could land inside somebody else's statement
`SQ_CLOSE` and `SQ_RELEASE` act on whatever statement is current; neither names one. Two consequences.
A cursor finalizer that can't take the wire lock queues its cleanup for the next operation to flush — but that queue was flushed before *every* PDU, and a finalizer queues precisely because another thread is mid-statement. The flush landed inside that thread's own statement and released it (`-208` before the first fetch, `-267` between fetch batches). Flushing only at a statement boundary is both correct and sufficient.
And the finalizer's own lock probe couldn't see its own thread. `RLock.acquire(blocking=False)` grants a reentrant acquire to the owner, so when GC fired on a thread that was mid-statement, the finalizer concluded it had exclusive access and sent `CLOSE`/`RELEASE` into the running query. Refcounting hides this — a dropped cursor is freed at the drop — but a cursor caught in a reference cycle waits for a collection, and cycles are ordinary. Any traceback holding a cursor makes one.
A stale queue entry was separately fatal: the leftover `CLOSE` draws `-267`, an `OperationalError`, which was treated as a dead wire and force-closed a perfectly healthy connection.
### The async layer borrowed the whole process's thread pool
Every blocking call went through `asyncio.to_thread`, which runs on the event loop's default executor — sized `min(32, cpu_count + 4)`, so six threads on a two-CPU container.
`asyncio.to_thread` cannot interrupt a worker, so a cancelled await leaves the thread running the wire call until the read timeout. Cancellation is ordinary in a web app; a client disconnect cancels the request task. Measured: six cancelled calls against a six-worker default executor starve an unrelated `to_thread` indefinitely, and with the executor held, **four concurrent driver queries never ran at all**. Pool concurrency was capped by the same unrelated number — `max_size=20` on a two-CPU box ran six queries at a time.
Each connection now owns one thread. That's the right size rather than a compromise, since the connection serializes every wire operation on its own lock anyway, and it avoids a deadlock a shared pool-sized executor invites.
### Two readers shared one stream without agreeing
`IfxSocket` owns a read-ahead buffer that the buffered reader fills, while the login path and the connection-level drain read straight from the socket without looking at it. Buffered bytes would be skipped, and skipped bytes in a length-framed protocol don't announce themselves.
Nothing triggers it today, because the server sends one response per request. That's a property of the traffic, not of the code — pipelined `executemany` already puts several responses in flight, and the buffer is connection-scoped precisely so read-ahead *can* cross response boundaries. The two paths now agree by construction.
Also here: `fill_recv_buf` believed whatever byte count it was handed, and that count is almost always a length field straight off the wire — a garbage `0x7FFFFFFF` read as a 2 GB request and sat in `recv` until the read timeout. It now refuses above `IFX_MAX_READ_BYTES` (256 MiB default) with an error naming the number.
### Verified
**457** integration tests on 15.0.1.0.3DE and 14.10.FC7W1DE, **456** on 12.10.FC12W1DE (one skip — 12.10 has no CTEs) — up from 414. Every fix has at least one test that fails against the previous release.
## 2026.09.01 — TLS traffic fuzzed; no bugs found
Tests and docs only — no behaviour change. TLS was the last untested surface, and it came back clean.
### Why it needed testing separately
Previous TLS coverage stopped at the handshake. Everything after it ran only over plain sockets, and `SSLSocket.recv` is not `socket.recv`: it returns at most one TLS record's worth of plaintext however much you ask for, it can return fewer bytes than are available, and plaintext buffered inside the SSL object is invisible to the OS. The Phase 39 buffered reader asks for up to 64 KB per call and loops until satisfied — that loop is the thing which has to be right, and nothing in the plain-socket suite put the same pressure on it.
`tests/test_tls_traffic.py` runs real SQLI traffic through a TLS-terminating proxy: the framing-bug types end-to-end, payloads at 1 / 4096 / **16383 / 16384 / 16385** / 32000 bytes (straddling the ~16 KB TLS record boundary), 500- and 5000-row bulk fetches, error recovery, concurrent TLS sessions, and three negative cases — TLS client against a plaintext port, plaintext client against a TLS port (must raise rather than hang), and a verifying context correctly rejecting a self-signed certificate.
**All clean on all three servers.** The buffered reader handles `SSLSocket` semantics correctly.
### Scope, stated plainly
The proxy supplies the TLS half, so this exercises the driver's TLS path — the half we own. It does **not** exercise IBM's server-side TLS listener. Setting one up on the developer-edition image was attempted and abandoned: Informix 15 wants a PKCS#12 keystore (`onkstash` takes a `.p12`, not the older CMS `.kdb`), and the engine kept rejecting the stash with `GSK_ERROR_BAD_KEYFILE_PASSWORD` even with a keystore GSKit itself could open. That half is IBM's code; everything below `ssl.wrap_socket` is identical either way.
### Verified
**414/414** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 399.
With this, every surface has been fuzzed: type framing, fetch batching, error recovery, cursor lifecycle, threads, pooling, transactions, async cancellation, `executemany` partial failure, scrollable cursors, smart LOBs, and TLS.
## 2026.08.31.3 — An encoding failure inside `executemany` leaked the statement
The last of the untested surface: pipelined `executemany`, scrollable cursors, and smart LOBs. One bug, in the first of those.
### The bug
`executemany` builds all its BIND+EXECUTE PDUs *after* the PREPARE and before anything is drained — that batching is what makes the pipeline fast. If encoding a row raises there (a value the connection's codec cannot represent), the exception escaped the whole block **without sending the RELEASE**, leaking the prepared statement. The next PREPARE collided with it and every later call on that connection failed with an error pointing at the previous SQL.
Same failure as `2026.08.31.2`, in a sibling path. `_execute_dml_with_params` has guarded this exact case for the single-row path for a long time; the pipelined path was simply never given the same treatment. That is the recurring shape of these: a hazard understood in one place and not carried across to the code next to it.
### What was already sound
Worth recording, because it's the larger part of the result:
- **`executemany` constraint failures.** Duplicate-key and NOT NULL violations at the first, middle, and last position of batches from 2 to 1000 rows all recover cleanly, and `COUNT(*)` always agrees with a full fetch. The pipelined drain-N-responses invariant holds under partial failure.
- **Scrollable cursors.** `fetch_first` / `fetch_last` / `fetch_prior` / `fetch_relative` / `fetch_absolute` are correct at 0, 1, 2, 5, 50 and 300 rows, including off both ends (`None`, not a wrap or a crash) and a full forward walk after arbitrary positioning. Twenty abandoned scroll cursors leak nothing.
- **Smart LOBs.** Round-trip at 0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535 and 65536 bytes — straddling the 4096-byte `SQ_FILE` chunk and the 64K mark — plus recovery from failed reads.
### Verified
**399/399** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 356.
### Honest note on what is still not fuzzed
TLS is covered by handshake tests against a self-signed local socket, which exercises the transport wrapper but not a real Informix TLS listener (that needs server-side keystore and `onconfig` SSL setup we don't have in the test containers). The SQLI layer above the socket is identical either way and is now heavily fuzzed, so the residual risk is confined to the handshake itself.
## 2026.08.31.2 — A failed statement killed the connection; cancelled acquires starved the pool
Two more found by fuzzing rather than by users, this time outside the type system. Both are ordinary-path bugs that a happy-path test cannot reach.
### A failed statement was never released
Successful DML sent `PREPARE → EXECUTE → RELEASE`. **Failing DML sent `PREPARE → EXECUTE` and stopped.** The prepared statement stayed allocated server-side, collided with the next `PREPARE`, and from then on every call on that connection returned a nonsense error — `-255 "Not in transaction"` under autocommit, `-285` otherwise — whose reported offset pointed back at the *failed* SQL rather than the statement that actually failed.
The practical shape of this: **one duplicate-key violation bricked the connection.** An `INSERT` that trips a unique constraint is about the most routine error an application can hit — every "insert if not exists" pattern produces them — and afterwards nothing on that connection worked again.
The docstring on the parameterised path already described this exact hazard and guarded the *parameter-encoding* failure. It just never covered the case where EXECUTE itself failed. Fixed in all three paths (DML, parameterised DML, and SELECT, whose fetch loop had the same gap and whose GC-time finalizer only covers scrollable cursors).
### Cancelling a pool acquire leaked the connection permanently
`asyncio.to_thread` cannot interrupt its worker. When a task awaiting `pool.acquire()` was cancelled *while the worker was still blocked waiting for a free connection*, the worker went on to succeed and hand back a connection **nobody owned** — checked out, never returned. Each occurrence cost one pool slot until the pool was dead.
This is not a theoretical race for anyone serving HTTP. A client disconnecting cancels the request task, and under load those cancellations land precisely while waiting for a connection. The pool dies one slot at a time, only under load, and the eventual symptom (`PoolTimeoutError`) points nowhere near the cause.
`acquire()` now shields the inner future and, if the caller goes away, returns whatever the worker produced to the pool. `add_done_callback` fires immediately on an already-resolved future, so "the worker finished a moment before the cancellation" is the same code path rather than a separate race.
### What the harness looked for
Three areas the type-matrix fuzzer cannot see:
- **Fetch batching.** `NFETCH` is a 4096-*byte* budget, so rows per batch moves with row width. Row counts from 0 to 1025 across narrow/medium/wide schemas, every fetch style (`fetchall`, `fetchone` loop, `fetchmany` at several sizes, iteration) required to agree exactly.
- **Error recovery.** Eight error classes, repeated, each followed by a known-good query on the same cursor *and* a fresh one. Repetition matters: a leak accumulates.
- **Concurrency.** Threads sharing one connection, pooled threads that deliberately fail before releasing, transaction isolation across pool checkouts, and async cancellation. Every worker reads back a value only it supplied, so a crossed wire fails on *data* even when nothing raises.
### Two harness false positives, recorded because they're easy to re-trip
Informix **silently truncates** over-long strings — a 500-character value into `VARCHAR(8)` stores 8 characters and raises nothing. Verified that a literal SQL insert behaves identically, so the driver matches the server; the test expectation was wrong, not the code.
Informix also **rejects a bare `?` in a projection** (`SELECT ? FROM t` is a syntax error — no type to infer). `SELECT ?::INT FROM t` works. Worth knowing before concluding the driver mishandles parameters.
### Verified
**356/356** integration tests on each of 15.0.1.0.3DE, 14.10.FC7W1DE, and 12.10.FC12W1DE — up from 326.
## 2026.08.31.1 — Three more bugs, found by a fuzzer instead of a user
Six framing bugs had reached users across three reports. Rather than wait for a seventh, this release adds a harness built specifically to find that class of bug — and it immediately found three more, one of which **hangs the connection**.

View File

@ -27,7 +27,7 @@ Imports as `informix_db` (the distribution name is `informix-driver` because the
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
**Test coverage:** 400+ tests across unit / integration / benchmark suites. The integration suite passes 326/326 against **each** of Informix 12.10, 14.10, and 15 — `make test-matrix` runs all three.
**Test coverage:** 400+ tests across unit / integration / benchmark suites. The integration suite passes 414/414 against **each** of Informix 12.10, 14.10, and 15 — `make test-matrix` runs all three.
## Quick start
@ -154,9 +154,9 @@ All three tested against the official IBM developer-edition Docker images, full
| Server | Image | Integration suite |
|---|---|---|
| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **326 / 326** |
| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **326 / 326** |
| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **326 / 326** |
| **15.0.1.0.3DE** | `icr.io/informix/informix-developer-database` | **414 / 414** |
| **14.10.FC7W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
| **12.10.FC12W1DE** | `ibmcom/informix-developer-database` | **414 / 414** |
Reproduce the whole matrix:

View File

@ -13,7 +13,7 @@
</h1>
<p class="ifx-hero__lede">
Every other Informix driver wraps IBM's C SDK or the JDBC JAR. We weren't into that.
So we read the protocol and wrote it ourselves — PEP 249, sync + async, pooled, TLS.
So we read the protocol and wrote it ourselves: PEP 249, sync and async, pooled, TLS.
Within 10% of IBM's own C driver on bulk fetches, <strong>1.6× faster</strong> on
bulk inserts. No compile step. No <code>LD_LIBRARY_PATH</code> ritual. No
<code>libcrypt.so.1</code> from 2018.

View File

@ -1,6 +1,6 @@
---
title: Architecture overview
description: How the layers stack — socket, framing, codec, resultset, cursor, connection, pool.
description: How the layers stack, from socket through framing, codec, resultset, cursor, connection, and pool.
sidebar:
order: 2
---
@ -28,13 +28,13 @@ The driver is six layers, each with a single responsibility, each testable in is
The lowest layer. Wraps `socket.socket` with a connection-scoped read buffer (Phase 39). One `recv(64K)` per ~64 KB of incoming data; parsers read into the buffer via `struct.unpack_from(buf, offset)` rather than slicing copies.
Everything above this layer is `bytes` and `bytearray` arithmetic no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
Everything above this layer is `bytes` and `bytearray` arithmetic, with no syscalls except through `IfxSocket.read_exact(n)` and `IfxSocket.write_all(buf)`.
See [The buffered reader →](/explain/buffered-reader/) for why the buffer lives here and not on the parser.
## Protocol / PDU framing
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean* only how to read and write the byte shapes.
`_protocol.py` reads and writes SQLI PDUs. Each PDU is parsed into a typed Python representation: `SqInfo`, `SqVersion`, `SqTuple`, `SqId`, etc. The framing layer doesn't know what the PDUs *mean*, only how to read and write the byte shapes.
The PDU types and their fields were reverse-engineered from three sources:
@ -44,9 +44,9 @@ The PDU types and their fields were reverse-engineered from three sources:
## Codec / Per-column readers
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types — see [SQL ↔ Python types](/reference/types/) for the full table.
`converters.py` and `_resultset.py` together. The codec layer maps Informix SQL types to Python types. See [SQL ↔ Python types](/reference/types/) for the full table.
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]` no per-column dispatch overhead.
Phase 37 introduced **per-column reader strategy**: at PREPARE time, the driver builds a list of decoder functions (one per column) keyed by SQL type. At fetch time, decoding a row is `[reader(payload) for reader in column_readers]`, with no per-column dispatch overhead.
Phase 38 went further with `exec()`-based codegen: for the hottest tables, the driver generates a flat decoder function with all readers inlined and dispatch decisions baked in. The generated function is the equivalent of unrolling the per-column dispatch into straight-line code.

View File

@ -1,6 +1,6 @@
---
title: Async strategy
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async and what that costs.
description: Why informix-driver wraps a sync core in a thread pool instead of going fully async, and what that costs.
sidebar:
order: 4
---
@ -19,13 +19,13 @@ Three options for adding async support to a sync database driver:
2. **Thread-pool wrapping.** Keep the sync core. Wrap each public method with `loop.run_in_executor()`. ~250 lines of code, sync tests still apply, no protocol-layer changes.
3. **Dual implementations.** Maintain two parallel code paths — one sync, one async. Most code duplicated. Worst of both worlds.
3. **Dual implementations.** Maintain two parallel code paths, one sync and one async. Most code duplicated. Worst of both worlds.
We picked option 2.
## Why option 2 was the right call
For typical database workloads — request-scoped connections, mostly waiting on I/O — the practical difference between option 1 and option 2 is small:
For typical database workloads, meaning request-scoped connections that mostly wait on I/O, the practical difference between option 1 and option 2 is small:
- **Latency**: option 1 has a slight edge (no thread context switch), but the difference is dwarfed by the actual database round-trip (~80 µs LAN, ~ms WAN). For a single query, option 2 adds ~510 µs of executor overhead.
- **Throughput under concurrency**: option 1 wins when you have N coroutines on M physical cores with M < N. The thread pool needs to context-switch between threads; the async loop just runs the next coroutine. For 10100 concurrent FastAPI requests on a 4-core box, this difference is small.
@ -39,7 +39,7 @@ The honest costs:
- **One worker thread per concurrent in-flight query.** With 100 concurrent queries, you have 100 threads. This is fine for I/O-bound work (Python releases the GIL during socket reads) but doesn't scale beyond a few hundred concurrent queries on a single process.
- **Thread-pool sizing matters.** The default executor size (5 × CPU count) is fine for most workloads. For high-concurrency workloads, you may want a larger executor.
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe — cancelled workers cannot leak onto recycled pool connections — but the underlying syscall does still complete.
- **Cancellation requires thought.** A cancelled `await cur.execute()` cancels the coroutine, but the worker thread continues running until the syscall returns. The connection is marked dirty until then. Phase 27 made this safe, in that cancelled workers cannot leak onto recycled pool connections, but the underlying syscall does still complete.
## What it doesn't cost

View File

@ -7,7 +7,7 @@ sidebar:
import { Aside } from '@astrojs/starlight/components';
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39 — a connection-scoped buffered reader — closed it from 2.4× to ~1.051.15× in about thirty minutes of code plus ten minutes of architectural debugging.
The bulk-fetch gap against IfxPy stayed stubbornly at ~2× from Phase 36 through Phase 38. Two phases of codec optimization shrank it by a few percent each. Phase 39, a connection-scoped buffered reader, closed it from 2.4× to ~1.051.15× in about thirty minutes of code plus ten minutes of architectural debugging.
This page is about both the technical change and the failure mode that hid the win for two phases.
@ -26,7 +26,7 @@ The headline "I/O dominated" was true. The interesting half is the breakdown of
- Actual `recv()` syscalls: ~153 ms
- Python wrapper overhead: ~400 ms
That ~400 ms was our own buffer abstraction a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
That ~400 ms was our own buffer abstraction: a `read_exact` loop that called `recv()` per fragment, reassembled fragments via `bytes.join`, and traversed two layers of cursor wrappers per call. For 100,000 rows that's **451,402 calls to `read_exact`**, each one paying Python wrapper cost the kernel didn't cause.
The kernel was doing maybe 2530 ms of work. The other 130 ms of the gap-vs-IfxPy was friction we had introduced ourselves.
@ -63,21 +63,21 @@ Result: **one `recv()` per ~64 KB of incoming data**, not per field.
The natural thing to call this is "BufferedSocketReader". The natural thing to do is put the bytearray on the reader. That's what I did first.
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken` — process blocked in `recv()` waiting for bytes that weren't coming.
Then `test_executemany_1000_rows` hung. The kernel stack via `cat /proc/PID/wchan` said `wait_woken`, meaning the process was blocked in `recv()` waiting for bytes that weren't coming.
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes — 200 bytes for response 1, 400 bytes for response 2 — it consumed bytes for response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
The bug was foreseeable, and it was architectural rather than implementational. Phase 33's pipelined `executemany` sends N BIND+EXECUTE PDUs back-to-back and drains responses afterward. Each cursor read constructs a *new* reader instance. When my reader did `recv(64K)` and pulled in 600 bytes, 200 for response 1 and 400 for response 2, it consumed bytes belonging to response 2 *and then was destroyed*. The next reader called `recv()`, the kernel buffer was empty, and we waited forever for bytes the kernel had already given to a dead reader.
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket` (the connection-scoped wrapper) — readers are short-lived parser-views, the buffer outlives them.
The fix moved the buffer one level down. The bytearray and offset cursor live on `IfxSocket`, the connection-scoped wrapper. Readers are short-lived parser-views, and the buffer outlives them.
```python
# WRONG (first pass) buffer scoped to reader
# WRONG (first pass): buffer scoped to reader
class BufferedSocketReader:
def __init__(self, sock):
self.sock = sock
self.buf = bytearray() # ← dies with the reader
self.offset = 0
# RIGHT (Phase 39) buffer scoped to connection
# RIGHT (Phase 39): buffer scoped to connection
class IfxSocket:
def __init__(self, sock):
self.sock = sock
@ -122,7 +122,7 @@ The buffered reader ships **enabled by default** in version 2026.05.05.12. To op
IFX_BUFFERED_READER=0 python my_app.py
```
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime close and reopen the pool to flip behavior.
The flag is read once at connection construction. Existing connections in a pool aren't affected by changing the env at runtime, so close and reopen the pool to flip behavior.
<Aside type="note">
The flag exists to make A/B measurement easy. There's no expected reason to disable it in production. If you hit a workload where the buffered reader is slower, that's a bug and we'd like to know.
@ -132,9 +132,9 @@ The flag is read once at connection construction. Existing connections in a pool
The general pattern: **what's visible gets optimization attention; what's invisible gets written off as irreducible**.
The codec is visible there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
The codec is visible: there's a loop, a `_decode_varchar` function, a `struct.unpack` call. You can read the inner loop and reason about it. Phases 37 and 38 attacked it, both got modest wins.
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads — `read_short`, `read_int`, `read_exact(payload_size)` — are one-liners. What could be slow about that?
The I/O machinery looked invisible. `_socket.read_exact` is eight lines. The cursor's `_SocketReader` wrapper is twelve. The framing reads (`read_short`, `read_int`, `read_exact(payload_size)`) are one-liners. What could be slow about that?
It was carrying ~30% of total wall time. Two phases of changelogs implicitly blamed "the protocol" for the remaining gap. The actual culprit was a few lines of `bytes.join` in a wrapper from Phase 1 that nobody had revisited.
@ -142,6 +142,6 @@ The lesson is small and easy to state: a profile turns vibes into an attack surf
## Read more
- **[Architecture overview →](/explain/architecture/)** where the buffered reader sits in the layer stack.
- **[Phase log →](/explain/phase-log/)** the full progression from Phase 1 through Phase 39+.
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)** Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.
- **[Architecture overview →](/explain/architecture/)**: where the buffered reader sits in the layer stack.
- **[Phase log →](/explain/phase-log/)**: the full progression from Phase 1 through Phase 39+.
- **["The 156 Milliseconds I'd Been Hand-Waving About"](https://ryanmalloy.com/collaborations/the-156-milliseconds-i-d-been-hand-waving-about/)**: Claude's reflection on the session in which Phase 39 shipped, including the two pushbacks that triggered the work.

View File

@ -43,11 +43,11 @@ The driver was built across 39+ phases, each with a focused scope and a decision
| 23 | Health checks | Pool validates idle connections before return |
| 24 | Statement caching | Per-connection prepared-statement cache |
| 25 | Fast-path call (`SQ_FPROUTINE`) | Direct UDF/SPL invocation, bypassing PREPARE |
| 26 | **CRITICAL** | Pool returned connections with open transactions — fixed |
| 26 | **CRITICAL** | Pool returned connections with open transactions (fixed) |
| 27 | **CRITICAL** | Per-connection wire lock + async cancellation safety |
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync — fixed |
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise — fixed |
| 30 | Hardening pass | 5 medium-severity audit findings all closed |
| 28 | **HIGH** | `_raise_sq_err` bare-except masking wire desync (fixed) |
| 29 | Cursor finalizers | Server-side resource leak on mid-fetch raise (fixed) |
| 30 | Hardening pass | 5 medium-severity audit findings, all closed |
After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver is production-ready.
@ -67,21 +67,84 @@ After Phase 30: **0 critical, 0 high, 0 medium audit findings remain.** Driver i
The Phase 3739 trajectory is documented in detail at [The buffered reader →](/explain/buffered-reader/), including the architectural mistake the first pass got wrong.
## Field reports and systematic review (2026.082026.09)
Phase 30's audit found nothing left. Then real schemas arrived, and the audit's clean bill of health turned out to be a statement about the questions it had asked rather than about the driver.
A field report from a user running Informix 12 surfaced three type-framing bugs in a single afternoon: `INT8` / `SERIAL8` never decoded at all, `NCHAR` losing its first character, and `BOOLEAN` corrupting every column after it. Fuzzing the type matrix found more, and each one had the same shape as the last.
`2026.09.02` went back over the driver looking for that shape rather than for new symptoms, and found eleven bugs.
| Area | What was wrong |
|---|---|
| Row framing | Nothing ever checked that a row consumed its own payload. Fourteen framing bugs had reached users, every one detectable for free |
| Smart LOBs | Read as a flat 72-byte field when they occupy 149, shifting every column after a non-final `BLOB` |
| Composites | A NULL collection skipped its length field, the same bug as NULL `LVARCHAR` twelve lines away |
| Transactions | `rollback()` after a SQL `BEGIN WORK` sent nothing and reported success |
| Classification | Comments, CTEs and parenthesized selects were run as DML and failed with `-260` |
| Scrollable cursors | A second statement on the connection got `-285` and destroyed the cursor too |
| Statement release | Two of eight exits leaked a failed statement, bricking the connection |
| Cursor finalizers | Cleanup could land inside another statement, and the lock probe was blind to its own thread |
| Async | The whole layer shared the process-wide thread pool, so cancellations starved it |
| Socket | Two readers on one stream agreed only by accident of how the server replies |
The pattern behind almost all of them: a hazard that was understood and guarded at the site where it was first observed, rather than at the abstraction that owned it. The guard then never travelled to its siblings. Four separate hand-written copies of the same UDT envelope, three of them wrong. Six hand-rolled copies of the same statement-release cleanup, two missing entirely.
Two of the fixes came from asking the server instead of guessing. `statement_type` and `statement_id` had both been parsed out of every DESCRIBE response into a metadata dict that nothing read, while the code that needed them inferred the answer from the first word of the SQL and got it wrong five ways.
The cheapest fix was also the most valuable. Asserting that a row decoder lands exactly on the end of its payload costs one integer comparison, and it would have caught all fourteen framing bugs at the byte where each happened rather than three releases later in somebody's result set.
Test count went from 241 to 457 across three server versions over this stretch.
### 2026.09.03
Two more, one of each kind.
The `:N` placeholder rewriter was a bare regular expression, so it
rewrote the inside of string literals: `'http://host:8080/x'` was stored
as `'http://host?/x'`. Same family as the statement-classification bug
above, and the last place the driver inferred meaning from SQL text
instead of handling it. Replaced with a scanner using Informix's own
lexical rules, measured rather than assumed. One of those rules would
have been got wrong from habit: a backslash escapes nothing, so `'a\'b'`
is an unterminated string and draws `-282`. A scanner written to
Postgres reflexes desyncs there and corrupts everything after it.
`row_factory=Row` adds column-name and attribute access, on request from
a user coming from `pyodbc`. Opt-in, because it costs about 9% on bulk
fetch and that is not a trade to make for someone who never reads a
column by name.
Testing the second one found the pattern again, this time in code
written an hour earlier. The names a column could shadow were
hand-listed as `count` and `index`, the two `tuple` methods; then `keys`,
`_asdict` and `_fields` were added to the same class and the list was
never revisited. A column called `keys` returned a bound method, in
silence. The fix was to compute the set from the class rather than
lengthen the list, and to parametrize the test over that computed set so
it grows with the class.
The interesting part is where testing found it. A differential test
across twenty columns covering every awkward type, populated and fully
NULL, found nothing. The bug was in six lines of name bookkeeping that
looked obviously correct, and had been correct when written. Anything
that must stay in sync with a class by hand eventually will not.
## Notable architectural pivots
The decision log calls out four moments where the obvious choice would have been wrong:
1. **Phase 10/11** — abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
2. **Phase 16** — thread-pool async instead of full async refactor. ~88% less code, same FastAPI surface.
3. **Phase 27** — adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
4. **Phase 39** — buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
1. **Phase 10/11**: abandoning `SQ_FPROUTINE` + `SQ_LODATA` for `SQ_FILE` intercept. Smaller, simpler, same correctness.
2. **Phase 16**: thread-pool async instead of a full async refactor. ~88% less code, same FastAPI surface.
3. **Phase 27**: adding a per-connection wire lock instead of relying on PEP 249's "don't share connections" advice. Made accidental sharing safe rather than catastrophic.
4. **Phase 39**: buffer on the connection, not on the reader. Got it wrong on the first pass; the bug surfaced as a hang on pipelined `executemany`. Fixed in ten minutes once the architectural mistake was named.
## What's next
The roadmap (loose, not committed):
- **Phase 40+ (codec)**: Numpy-backed bulk decode for homogeneous columns. ~5× speedup target on analytical workloads.
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python" — gated behind a build flag.
- **Phase 4x (protocol)**: Optional Cython acceleration for the codec hot loop. Would compromise "pure Python", so it would sit behind a build flag.
- **Phase 5x (API)**: Native `callproc` with named parameters, IBM-specific scrollable cursor extensions for full IfxPy parity.
The phase log is updated as work lands. The repo's [`CHANGELOG.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/CHANGELOG.md) is the source of truth for shipped changes.

View File

@ -26,14 +26,14 @@ The order-of-magnitude intuition: pure-Python is ~2× slower than C-bound for **
The benefits are mostly deployment, not performance:
- **50 KB wheel** installable in a slim Docker image without a build toolchain.
- **No `libcrypt.so.1`** works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
- **Python 3.103.14** no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
- **Type annotations everywhere** `py.typed` flag, full coverage in mypy / pyright.
- **Auditable codepaths** every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
- **Async without `run_in_executor`** native `async def` API, FastAPI-compatible.
- **50 KB wheel**: installable in a slim Docker image without a build toolchain.
- **No `libcrypt.so.1`**: works on Arch, Fedora 35+, RHEL 9, and any modern Linux.
- **Python 3.103.14**: no minor-version-specific C extension breakage. We've shipped on the day each new Python released.
- **Type annotations everywhere**: `py.typed` flag, full coverage in mypy / pyright.
- **Auditable codepaths**: every byte that enters or leaves a socket goes through Python code you can read. No "the C extension does it" excuses.
- **Async without `run_in_executor`**: native `async def` API, FastAPI-compatible.
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1` those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
For most real workloads, deployment friction matters more than 2 µs/row. The 92 MB OneDB tarball, the four `LD_LIBRARY_PATH` entries, the absent `libcrypt.so.1`: those costs are paid every time you deploy. The 2 µs/row codec gap is paid once per row, and only if your workload is read-heavy enough for it to dominate.
## Where the ceiling sits
@ -48,7 +48,7 @@ Five fields × ~250 ns/field + ~250 ns overhead = ~1.5 µs. We're at ~2.0 µs wh
Strategies for closing further:
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim there'd be a build step.
- **Cython / mypyc compilation.** Could shave 30-50% off the codec hot loop. Would compromise the "pure Python" claim, because there'd be a build step.
- **Bytecode optimization via `exec()`-codegen** (the Phase 38 approach). Marginal further wins; we've already extracted most of what's available.
- **Numpy-backed bulk decode** for homogeneous columns. Promising for analytical workloads. ~5× speedup possible for `SELECT col FROM huge_table` over the current per-row approach. Probably Phase 41+.
@ -58,6 +58,6 @@ For I/O-bound workloads we're already at the ceiling. The buffered reader closed
Pure-Python costs us ~515% on bulk-fetch workloads and zero (or favorable) on everything else. The deployment, async, and modern-Python wins are large and don't depend on workload.
If the codec gap matters for your case — analytical reporting against a wide table, pulling millions of rows in a single SELECT — IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
If the codec gap matters for your case, meaning analytical reporting against a wide table or pulling millions of rows in a single SELECT, IfxPy is probably the right tool today. If you're doing transactional or bulk-load work, FastAPI services, or any deployment where IBM's C SDK is friction, `informix-driver` is the right tool.
The driver chose the goal*first pure-socket Informix driver in any language* over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.
The driver chose the goal, *first pure-socket Informix driver in any language*, over the local optimum. Phase 37 onward is a sustained effort to make that choice cost as little as possible.

View File

@ -1,13 +1,13 @@
---
title: The SQLI wire protocol
description: A short tour of Informix's SQLI protocol — PDU framing, the handshake, statement execution, fetch.
description: A short tour of Informix's SQLI protocol, covering PDU framing, the handshake, statement execution, and fetch.
sidebar:
order: 1
---
import { Aside } from '@astrojs/starlight/components';
SQLI is Informix's wire protocol — the same protocol IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
SQLI is Informix's wire protocol, the same one IBM's CSDK and JDBC driver speak. It's a binary, length-prefixed PDU stream over a single TCP connection.
This page is a short tour. The byte-level reference (with hex annotations) lives in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md) in the repo.
@ -71,14 +71,14 @@ The full lifecycle for `SELECT id FROM users WHERE id = ?`:
← SQ_ID
```
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts — see [Bulk inserts](/how-to/executemany/).
For pipelined `executemany`, the driver sends `SQ_OPEN`+`SQ_FETCH` (or `SQ_BIND`+`SQ_EXEC`) for all N rows back-to-back without waiting for responses, then drains all responses at the end. This is what gives the 1.6× win over IfxPy on bulk inserts. See [Bulk inserts](/how-to/executemany/).
## Smart-LOB transfer
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
`SQ_FILE` (0x62) is a self-contained PDU type that carries chunks of BLOB/CLOB data. It's used by Informix's `lotofile` and `filetoblob` server functions. The driver intercepts these PDUs at the wire level and reassembles them client-side, with no `SQ_FPROUTINE` / `SQ_LODATA` machinery needed.
This was the architectural pivot in [Phase 10/11](/explain/phase-log/) that made smart-LOBs work end-to-end in pure Python. Reading and writing GB-sized BLOBs goes through the same socket as any other query.
<Aside type="note">
The protocol has many more PDU types than this page covers mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
The protocol has many more PDU types than this page covers, mostly variants for specific server features (PUT, GET-DESCRIPTOR, ROWDESC, DBINFO, COLLINFO, etc.). The complete list is in [`docs/PROTOCOL_NOTES.md`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/docs/PROTOCOL_NOTES.md), with hex captures for each.
</Aside>

View File

@ -56,7 +56,7 @@ async def get_user(user_id: int, conn = Depends(get_conn)):
## Cancellation
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean (transactions rolled back). You don't need to wrap anything in `try/finally`.
If a client disconnects mid-request, FastAPI cancels the task. `informix-driver` is cancellation-safe: the cancellation propagates cleanly, the in-flight worker is reaped, and the connection returns to the pool clean, with transactions rolled back. You don't need to wrap anything in `try/finally`.
<Aside type="note">
This is a Phase 27 invariant: async cancellation cannot leak running workers onto recycled connections. The earlier behavior was a `High` audit finding; the fix is a CI tripwire test that's been green every commit since.
@ -89,5 +89,5 @@ async def create_order(order: OrderIn, conn = Depends(get_conn)):
`HTTPException` is an `Exception`, so raising one inside the block rolls back before FastAPI turns it into a response.
<Aside type="note">
There is no `conn.transaction()` context manager on either the sync or async connection an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
There is no `conn.transaction()` context manager on either the sync or async connection, though an earlier version of this page showed one. Wrapping this in your own `@asynccontextmanager` helper is a few lines if you want the shorthand.
</Aside>

View File

@ -1,11 +1,11 @@
---
title: Optimize bulk SELECT
description: How the buffered reader works in practice — when it's on, when it isn't, how to A/B-measure your workload.
description: How the buffered reader works in practice, when it's on, when it isn't, and how to A/B-measure your workload.
sidebar:
order: 5
---
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything the bulk-fetch gap against IfxPy is now ~515% rather than ~140%.
The connection-scoped buffered reader (Phase 39) is **enabled by default** as of `2026.05.05.12`. For most workloads you don't need to touch anything, since the bulk-fetch gap against IfxPy is now ~515% rather than ~140%.
For the architectural rationale, see [The buffered reader →](/explain/buffered-reader/).
@ -33,7 +33,7 @@ For typical bulk-SELECT workloads expect a 3040% wall-time reduction. For wor
## When the speedup is largest
Workloads where every column read makes ~45 small `recv()` calls — i.e. tabular data, narrow rows, large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
Workloads where every column read makes ~45 small `recv()` calls, meaning tabular data, narrow rows, and large row counts. The buffered reader replaces N small `recv()` calls with one `recv(64K)` per ~64 KB of incoming data.
| Workload shape | Speedup |
|---|---:|

View File

@ -1,6 +1,6 @@
---
title: Run the dev container
description: IBM Informix Developer Edition in Docker — first-time setup, sbspace for smart-LOBs, common troubleshooting.
description: IBM Informix Developer Edition in Docker, covering first-time setup, sbspace for smart-LOBs, and common troubleshooting.
sidebar:
order: 8
---
@ -20,9 +20,9 @@ docker run -d --name informix-dev \
icr.io/informix/informix-developer-database:15.0.1.0.3DE
```
- **`9088`** clear-text SQLI listener
- **`9089`** TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
- **`--privileged`** required for the dev image's shared-memory tuning
- **`9088`**: clear-text SQLI listener
- **`9089`**: TLS-enabled SQLI listener (server-side cert is self-signed; use `tls=True` in dev)
- **`--privileged`**: required for the dev image's shared-memory tuning
The image takes ~90 seconds to initialize. Watch for `oninit running`:
@ -84,10 +84,10 @@ pytest -m integration
**"Connection refused" on 9088**: the image is still initializing. Wait for `oninit running` in the logs.
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing it always exists.
**Login succeeds, queries fail with `-329` (database does not exist)**: you're connecting to a database that hasn't been created yet. Use `database="sysmaster"` for ad-hoc testing, since it always exists.
**`-908` (system error / shared memory)**: the container needs `--privileged`. Restart with that flag.
<Aside type="tip">
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart the database survives container shutdown.
For a long-running dev environment, set `restart: unless-stopped` in a compose file. The image is well-behaved on restart, and the database survives container shutdown.
</Aside>

View File

@ -1,6 +1,6 @@
---
title: Bulk inserts (executemany)
description: How to bulk-load with executemany — and the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
description: How to bulk-load with executemany, including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
sidebar:
order: 4
---
@ -49,7 +49,7 @@ The default is `autocommit=False`, so this only catches you if you've explicitly
## Why it's faster than IfxPy
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row — for 10,000 rows on a 80 µs RTT, that's 800 ms of just waiting for ACKs.
IfxPy's `executemany` calls `IfxPy.execute(stmt, tuple)` internally per row. That's one round-trip per row, so for 10,000 rows on an 80 µs RTT it adds up to 800 ms of just waiting for ACKs.
Phase 33 changed our `executemany` to **pipeline** the BIND+EXECUTE PDUs:
@ -78,7 +78,7 @@ with conn:
cur = conn.cursor()
for batch in chunks(huge_iterator, 10_000):
cur.executemany("INSERT INTO logs ...", batch)
# one transaction, many batched executemany calls one commit at the end
# one transaction, many batched executemany calls, one commit at the end
```
10,000 rows per chunk is a reasonable default; the per-chunk Python memory cost is `~ N × bytes_per_row`. For 10k tuples of 5 small fields that's a few MB.
@ -95,7 +95,7 @@ new_id = cur.fetchone()[0]
`sqlca.sqlerrd1` is where Informix records the serial value from the most recent INSERT on this connection, so read it before running any other statement.
There is no `cursor.lastrowid` an earlier version of this page said there was.
There is no `cursor.lastrowid`, though an earlier version of this page said there was.
For batch inserts that need the IDs, the idiomatic pattern is:

View File

@ -27,11 +27,11 @@ conn = informix_db.connect(
)
```
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported (deliberately — they're a security and parsing footgun).
The `informix-driver` keyword-argument form is closer to `psycopg`/`asyncpg` shapes. Connection strings aren't supported, deliberately: they're a security and parsing footgun.
## The DB-API surface is the same
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, `close()` all behave per PEP 249.
`cursor()`, `execute()`, `fetchone()`, `fetchmany()`, `fetchall()`, `executemany()`, `description`, `rowcount`, and `close()` all behave per PEP 249.
The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `DatabaseError`, `DataError`, `OperationalError`, `IntegrityError`, `InternalError`, `ProgrammingError`, `NotSupportedError`.
@ -45,9 +45,39 @@ The exception hierarchy is identical: `Error`, `Warning`, `InterfaceError`, `Dat
- **Async API** (`from informix_db import aio`)
- **Connection pool** (`informix_db.create_pool` / `aio.create_pool`)
- **Type-safe annotations** `informix-driver` ships with `py.typed`
- **Type-safe annotations**: `informix-driver` ships with `py.typed`
- **Python 3.12+ support**
- **Pipelined `executemany`** — 1.6× faster than IfxPy's per-row implementation
- **Pipelined `executemany`**: 1.6× faster than IfxPy's per-row implementation
- **Rows addressable by name**: `row["col"]` and `row.col`, not just `row[0]`
## Reading rows by name
IfxPy gives you positional access and nothing else, which is why most
IfxPy codebases grow a helper like this:
```python
cols = [c[0].lower() for c in cur.description]
row_dict = dict(zip(cols, cur.fetchone()))
```
That gets you a dict and still no attribute access. Ask for `Row`
instead and all three work at once:
```python
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur.execute("SELECT config_key, config_value FROM settings")
row = cur.fetchone()
row[0], row["config_key"], row.config_key
```
Set on the connection, so it applies to every cursor from it. The
`.lower()` in the hand-written version is already a no-op, incidentally:
Informix folds unquoted identifiers to lower case.
It is opt-in because it costs about 9% on bulk fetch. See
[the API reference](/reference/api/#rows) for the numbers and the
edge cases.
## Migrating incrementally

View File

@ -1,6 +1,6 @@
---
title: Use the connection pool
description: Sync and async connection pools — sizing, timeouts, lifecycle, threading.
description: Sync and async connection pools, covering sizing, timeouts, lifecycle, and threading.
sidebar:
order: 2
---
@ -32,7 +32,7 @@ with pool.connection() as conn:
pool.close()
```
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically you never see a dirty connection from `pool.connection()`.
The context manager guarantees the connection returns to the pool on normal exit *and* on exception. Connections returned to the pool get rolled back automatically, so you never see a dirty connection from `pool.connection()`.
## Async pool
@ -56,16 +56,16 @@ async def main():
asyncio.run(main())
```
Same semantics, `async`-aware. Cancellation is cancellation-safe — a cancelled task does not leak an in-flight worker onto a recycled connection.
Same semantics, `async`-aware. Cancellation is safe here too: a cancelled task does not leak an in-flight worker onto a recycled connection.
## Sizing
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes" usually 816 for typical web/API services.
A reasonable starting point: `min_size = 2`, `max_size = (CPU cores) × 2`. Most Informix workloads are I/O-bound, so the right size is "enough to saturate the network plus some headroom for spikes", usually 816 for typical web/API services.
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`** — the server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
`max_size` should be **smaller than the server's `MAX_CONCURRENT_CONNECTIONS`**. The server fails new logins past its limit, and the pool will surface that as `OperationalError` after waiting `acquire_timeout`.
## Threading
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
PEP 249 says: connections should not be shared between threads. The pool gives each thread its own connection naturally, since `pool.connection()` returns a different connection each time and each one stays held until the context manager exits.
Phase 27 added a per-connection wire lock that makes accidental sharing safe (interleaved PDUs serialize correctly), but you should still give each thread its own connection. The lock is a backstop, not a license.

View File

@ -34,7 +34,7 @@ The `BLOB_PLACEHOLDER` token in the SQL marks where the BLOB data goes. Other pa
## Reading a CLOB
CLOBs use the same methods as BLOBs there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
CLOBs use the same methods as BLOBs, and there is no separate `read_clob_column`. `read_blob_column` returns `bytes` for both, so decode it with the encoding the column was written in:
```python
raw: bytes = cur.read_blob_column(

View File

@ -30,7 +30,7 @@ conn = informix_db.connect(
)
```
Bring-your-own context is the recommended production pattern — you get full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
Bring-your-own context is the recommended production pattern, giving you full control of certificate verification, hostname checking, ciphers, and TLS version pinning.
## Dev / self-signed: tls=True
@ -38,7 +38,7 @@ Bring-your-own context is the recommended production pattern — you get full co
informix_db.connect(host="127.0.0.1", port=9089, ..., tls=True)
```
`tls=True` is a convenience for development — it builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
`tls=True` is a convenience for development. It builds a default context with `check_hostname=False` and `verify_mode=CERT_NONE`. **Do not use this in production.**
## Server-side configuration
@ -48,7 +48,7 @@ The Informix server needs a TLS listener entry in `sqlhosts`:
informix_tls onsoctcp myhost 9089
```
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089` no configuration needed.
Plus a server-side keystore. The IBM Developer Edition Docker image ships with a TLS listener already enabled on `9089`, with no configuration needed.
<Aside type="tip">
If your connection hangs at the handshake, you've probably pointed at the non-TLS port (`9088` instead of `9089`). The non-TLS listener will accept the TCP connection but won't speak TLS, so the handshake stalls.

View File

@ -1,6 +1,6 @@
---
title: informix-driver
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket — no CSDK, no JVM, no native libraries.
description: Pure-Python driver for IBM Informix IDS. Speaks the SQLI wire protocol over a raw socket, with no CSDK, no JVM, and no native libraries.
template: splash
hero:
tagline: ''
@ -21,12 +21,12 @@ import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
<div class="ifx-feature">
<Icon name="seti:python" class="ifx-feature__icon" />
<h3>~10% behind IfxPy on bulk fetches</h3>
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics for now.</p>
<p>Phase 39's buffered reader closed the gap from 2.4× to within measurement noise of the C driver. The remaining ~10% is honest physics, for now.</p>
</div>
<div class="ifx-feature">
<Icon name="puzzle" class="ifx-feature__icon" />
<h3>50 KB wheel. Zero native deps.</h3>
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.103.14 including the versions IfxPy doesn't.</p>
<p>No 92 MB OneDB tarball. No <code>libcrypt.so.1</code> from 2018. No <code>LD_LIBRARY_PATH</code> ritual. Works on Python 3.103.14, including the versions IfxPy doesn't.</p>
</div>
<div class="ifx-feature">
<Icon name="sun" class="ifx-feature__icon" />
@ -36,7 +36,7 @@ import { Card, CardGrid, Icon } from '@astrojs/starlight/components';
<div class="ifx-feature">
<Icon name="approve-check" class="ifx-feature__icon" />
<h3>PEP 249, no surprises</h3>
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy — and threadsafe sharing through a per-connection wire lock.</p>
<p><code>connect()</code>, <code>Connection</code>, <code>Cursor</code>, <code>description</code>, <code>rowcount</code>, the full DB-API exception hierarchy, plus threadsafe sharing through a per-connection wire lock.</p>
</div>
<div class="ifx-feature">
<Icon name="document" class="ifx-feature__icon" />
@ -66,7 +66,7 @@ That's it. No `IBM_DB_HOME`. No DSN file. No `libcrypt.so.1`.
The existing tools were not my style.
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix` — all of them. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
Every other Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That means all of them: `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge **`informix-driver` is the first pure-socket Informix driver in any language**.
The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9). It needs four `LD_LIBRARY_PATH` entries. It needs `setuptools < 58`. And IfxPy itself is broken on Python 3.12+. For containerized deployments, ETL pipelines, FastAPI services, or anywhere a build toolchain on the runtime is friction, this driver is the alternative that didn't previously exist. Now it does.
@ -82,7 +82,7 @@ The OneDB CSDK is a 92 MB tarball. It needs `libcrypt.so.1` (deprecated 2018, mi
[Read →](/start/vs-ifxpy/)
</Card>
<Card title="The buffered reader" icon="information">
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1× and the architectural mistake the first pass got wrong.
How Phase 39 closed the bulk-fetch gap from 2.4× to ~1.1×, and the architectural mistake the first pass got wrong.
[Read →](/explain/buffered-reader/)
</Card>
<Card title="Architecture" icon="puzzle">

View File

@ -51,7 +51,7 @@ informix_db.connect(
| Method / property | Description |
|---|---|
| `cursor(scrollable=False)` | Returns a new `Cursor`. |
| `cursor(scrollable=False)` | Returns a new `Cursor`. Only one scrollable cursor may be open per connection; see below. |
| `commit()` | Commits the current transaction. |
| `rollback()` | Rolls back the current transaction. |
| `close()` | Closes the connection. Idempotent. |
@ -64,7 +64,7 @@ informix_db.connect(
`Connection` is also a context manager (`with informix_db.connect(...) as conn:`), which closes on exit.
There is no `conn.transaction()` helper and no `conn.autocommit` attribute — earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
There is no `conn.transaction()` helper and no `conn.autocommit` attribute. Earlier versions of this page listed both, and neither has ever existed. Set autocommit at connect time with `connect(autocommit=True)`, and manage transactions with `commit()` / `rollback()`:
```python
conn = informix_db.connect(..., autocommit=False)
@ -78,12 +78,67 @@ except Exception:
raise
```
:::caution[One statement per session]
Informix gives a session a single statement slot. A non-scrollable cursor never trips over this, because it materializes its rows and releases the statement before `execute()` returns. A **scrollable** cursor holds the slot open on purpose, so while one is open, any other statement on that same connection raises `ProgrammingError`:
```python
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT id, body FROM big_table ORDER BY id")
scroll.fetch_first()
other = conn.cursor()
other.execute("SELECT COUNT(*) FROM audit") # ProgrammingError
```
Close the scrollable cursor first, or use a second connection for the other query. Re-running `execute()` on the scrollable cursor itself is fine; it closes its own server-side cursor first.
Before `2026.09.02` this was not refused. The server answered the second statement with `-285` and destroyed the scrollable cursor as well, whose next fetch then returned `-267`, "the transaction has been rolled back". Two errors, neither naming the cause.
:::
:::note[Transaction control written as SQL]
`commit()` and `rollback()` are the supported way to end a transaction, but `cursor.execute("BEGIN WORK")` and its `COMMIT` / `ROLLBACK` counterparts work too, and the connection tracks them from `2026.09.02` onward.
That matters because it used not to. Under `autocommit=True`, a SQL `BEGIN WORK` opened a real transaction the connection never learned about, and since `rollback()` is guarded by that state, **it returned successfully having sent nothing**. The rows it was asked to discard survived, and a pooled connection went back into circulation still holding the transaction and its locks.
:::
:::note[Two version numbers, and why]
The login response carries Informix's *internal* protocol version, not the release you installed: 12.10 announces itself as `9.56`, 14.10 as `9.59`, and 15 as `15.0.1.0.3`. At the protocol level the two older releases really are 9.x servers, which is why all three speak an identical SQLI dialect.
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')` — one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
That string is available for free as `server_version_internal`. Because it reads as a wrong answer, `server_version` asks the server for its release with `DBINFO('version','full')`. That is one query on first access, cached for the life of the connection, and it falls back to the internal string rather than raising if no database is open.
:::
## Rows
Fetch methods return plain tuples by default. Pass `row_factory` to get rows that also answer to a column name and an attribute, the shape `pyodbc` and `mssql-python` provide:
```python
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur = conn.cursor()
cur.execute("SELECT tabid, tabname FROM systables")
row = cur.fetchone()
row[0] # 1
row["tabname"] # 'systables'
row.tabname # 'systables'
```
Set on the connection, so it applies to every cursor from it. A cursor can override it with `cur.row_factory = informix_db.Row`. Pools and the async API forward it unchanged.
`Row` subclasses `tuple`, so `row == (1, "systables")` is still true and anything treating rows as sequences keeps working. Slices return plain tuples, since a slice has no column map. `row.keys()` gives the column names and `row._asdict()` gives a dict.
:::note[Why it is opt-in]
On a 20,000-row five-column fetch: 37.2 ms with tuples, 40.8 ms with `Row`, about 9%. Supporting `row["name"]` means `__getitem__` becomes a Python method rather than C-level tuple indexing, which costs roughly 39 ns on every subscript.
Defaulting it on would move the [1.05-1.15x ratio against IfxPy](/start/vs-ifxpy/) to roughly 1.15-1.25x. That is a good trade for readable application code and a bad one for a bulk export that never looks at a column by name, so it is yours to make.
:::
Four behaviours worth knowing:
- Informix folds unquoted identifiers to lower case, so `SELECT Config_Key` is reachable as `row.config_key`. The `.lower()` people add by hand when building this themselves is already a no-op.
- Expression columns get server-generated names like `(count(*))`, which cannot be Python attributes. Those are reachable by subscript only.
- Duplicate column names resolve to the first occurrence, matching `pyodbc`. Positional access still reaches both.
- **A column beats a method of the same name.** `tuple` defines `count` and `index`; `Row` adds `keys`, `_asdict` and `_fields`. A column with any of those names wins, because otherwise it would hand back a bound method instead of a value. The methods stay reachable as `tuple.count(row, x)` and `Row.keys(row)`.
## Cursor
| Method / property | Description |
@ -105,7 +160,7 @@ That string is available for free as `server_version_internal`. Because it reads
| `rownumber` | Current 0-indexed position, or `None` before the first row. |
| `arraysize` | Default `fetchmany()` size. |
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute — earlier versions of this page listed all three and none have existed.
There are no `read_clob_column` / `write_clob_column` methods and no `lastrowid` attribute. Earlier versions of this page listed all three, and none have existed.
CLOBs go through the BLOB methods. `read_blob_column` returns `bytes` either way, so decode it yourself with the column's encoding:
@ -120,7 +175,7 @@ raw = cur.read_blob_column("SELECT txt FROM docs WHERE id = ?", (1,))
text = raw.decode("iso-8859-1")
```
For a server-assigned `SERIAL` after an INSERT, ask the server Informix exposes it through `DBINFO`:
For a server-assigned `SERIAL` after an INSERT, ask the server. Informix exposes it through `DBINFO`:
```python
cur.execute("INSERT INTO people (name) VALUES (?)", ("ada",))

View File

@ -1,6 +1,6 @@
---
title: Performance baselines
description: Single-connection benchmark results — codec, framing, end-to-end queries, vs IfxPy.
description: Single-connection benchmark results for codec, framing, end-to-end queries, and IfxPy.
sidebar:
order: 5
---

View File

@ -13,7 +13,7 @@ sidebar:
| `IFX_DEBUG_WIRE` | unset | When set to a truthy value, log wire-level PDU framing to stderr. Verbose; for debugging only. |
| `IFX_PROTOCOL_TRACE` | unset | When set to a path, write annotated wire captures to the file. |
| `IFX_DISABLE_PIPELINE` | unset | Disables pipelined `executemany` (Phase 33). Use only to A/B-measure. |
| `INFORMIXSERVER` | | Read by `connect()` if `server=` is not provided. |
| `INFORMIXSERVER` | *(none)* | Read by `connect()` if `server=` is not provided. |
Environment variables are read at connection construction. Changing them at runtime doesn't affect existing connections.
@ -64,4 +64,4 @@ informix_db.connect(
)
```
`CLIENT_LOCALE` is set automatically from `client_locale=` don't put it in `env=`.
`CLIENT_LOCALE` is set automatically from `client_locale=`, so don't put it in `env=`.

View File

@ -9,7 +9,7 @@ sidebar:
|---|---|---|
| `SMALLINT` / `INT` / `SERIAL` | `int` | Arbitrary precision on the Python side. |
| `BIGINT` / `BIGSERIAL` | `int` | 8-byte two's complement on the wire. |
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer — a different wire format from `BIGINT`, not an alias. See below. |
| `INT8` / `SERIAL8` | `int` | The *legacy* 64-bit integer, a different wire format from `BIGINT` rather than an alias. See below. |
| `FLOAT` / `SMALLFLOAT` | `float` | IEEE 754 double / single. |
| `DECIMAL(p,s)` / `MONEY` | `decimal.Decimal` | Exact precision preserved. |
| `CHAR` / `NCHAR` | `str` | Fixed width, space-padded; trailing spaces stripped on decode. |
@ -18,11 +18,11 @@ sidebar:
| `DATE` | `datetime.date` | |
| `DATETIME YEAR TO …` | `datetime.datetime` / `datetime.time` / `datetime.date` | The Python type depends on the field range. |
| `INTERVAL DAY TO FRACTION` | `datetime.timedelta` | |
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type `datetime.timedelta` can't represent year-month intervals. |
| `INTERVAL YEAR TO MONTH` | `informix_db.IntervalYM` | Custom type, because `datetime.timedelta` can't represent year-month intervals. |
| `BYTE` / `TEXT` (legacy in-row blobs) | `bytes` / `str` | |
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string **not** decomposed into fields. See below. |
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema **not** iterable. See below. |
| `BLOB` / `CLOB` (smart-LOBs) | `informix_db.BlobLocator` / `informix_db.ClobLocator` | Opaque server-side locators, 72 bytes. Read via `cursor.read_blob_column`, write via `cursor.write_blob_column`. |
| `ROW(…)` | `informix_db.RowValue` | Raw payload plus schema string, **not** decomposed into fields. See below. |
| `SET(…)` / `MULTISET(…)` / `LIST(…)` | `informix_db.CollectionValue` | Raw payload plus element schema, **not** iterable. See below. |
| `NULL` | `None` | |
## INT8 and SERIAL8 are not BIGINT
@ -34,16 +34,20 @@ Informix has two unrelated 64-bit integer types and they share nothing on the wi
| `BIGINT` / `BIGSERIAL` (type 52 / 53) | 8 bytes, two's complement, big-endian |
| `INT8` / `SERIAL8` (type 17 / 18) | 10 bytes, sign-magnitude, halves stored high-last |
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `n` with *identical* magnitude bytes and the sign in a leading word — decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
Both decode to plain Python `int`, so this only matters if you're reading the wire yourself. It's worth knowing that `INT8` stores `+n` and `n` with *identical* magnitude bytes and the sign in a leading word. Decoding it as a signed 64-bit integer looks correct for every positive value and is wrong for every negative one.
`INT8`/`SERIAL8` are common in schemas predating Informix 11.50, which is when `BIGINT` arrived.
:::caution[Upgrade if you use these types]
`2026.08.27` fixed three decoding bugs: `INT8`/`SERIAL8` weren't decoded at all and returned raw `bytes`; `NCHAR` lost its first character; and `BOOLEAN` corrupted every column after it.
`2026.08.31` fixed two more in `LVARCHAR` framing a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
`2026.08.31` fixed two more in `LVARCHAR` framing: a phantom pad byte on odd-length values, and a missing length field on NULLs. Either shifted **every column selected after the LVARCHAR**, producing wrong integers, strings missing their first character, or an outright `IndexError` on wide rows. The same release stopped `DATETIME` binds silently discarding sub-second precision.
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe — upgrade to `2026.08.31`.
`2026.09.02` fixed two more, both found by a new check that verifies each row consumed exactly its own payload. Smart LOBs were read as a flat 72-byte field when they actually occupy 149 bytes populated and 5 when NULL, so a `BLOB` or `CLOB` anywhere but the **last** column shifted every column after it. A NULL `SET` / `MULTISET` / `LIST` / `ROW` skipped its length field and did the same. That release also made a mismatch raise where it happens, naming the column and the byte delta, instead of handing back plausible-looking wrong values.
`2026.09.03` fixed one on the **write** path rather than the read path, which makes it the only entry here that affects data already stored. The `:N` placeholder rewriter was a regular expression and could not see a string literal, so it rewrote the inside of one: `'http://host:8080/x'` was stored as `'http://host?/x'`. Any `HH:MM` time, URL with a port, or `key:value` string was affected, whichever placeholder style you used. If you have written such values through this driver, they are worth checking.
If your schema has `LVARCHAR` columns, `2026.08.27` is not safe. If you select a `BLOB` or `CLOB` in any position other than last, or a nullable collection column, nothing before `2026.09.02` is safe. Upgrade.
:::
## LVARCHAR wire framing
@ -54,7 +58,7 @@ Worth knowing if you're reading captures. Every Informix server we've tested des
[1-byte null indicator][4-byte length][content]
```
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte both carry a length of zero.
No padding, and the length field is present even when the indicator says NULL. NULL and empty string differ *only* in that indicator byte, since both carry a length of zero.
## DATETIME field ranges
@ -68,17 +72,17 @@ Informix's `DATETIME YEAR TO X` is field-range typed. The Python type returned d
## Decimal precision
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly no float intermediate.
`DECIMAL` columns preserve their declared precision and scale through the codec. A `DECIMAL(10,2)` column with value `123.45` decodes to `Decimal("123.45")` exactly, with no float intermediate.
For binding `Decimal` values into INSERT/UPDATE, the driver uses the column's declared scale. Pass `Decimal` for exact values; `float` works but may cause rounding at the column scale.
## NULL
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
`NULL` is `None` in both directions. Use `IS NULL` / `IS NOT NULL` in SQL, because `WHERE x = ?` with `None` returns no rows even where `x IS NULL`.
## Type extensions
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count** — mirroring the server's own representation — with `years` and `remainder_months` available as derived properties:
`informix_db.IntervalYM` represents `INTERVAL YEAR TO MONTH`. It takes a **single total month count**, mirroring the server's own representation, with `years` and `remainder_months` available as derived properties:
```python
from informix_db import IntervalYM
@ -100,8 +104,8 @@ Negative intervals are supported; the sign lives on `months` and propagates to b
`informix_db.RowValue` and `informix_db.CollectionValue` are returned for `ROW`, `SET`, `MULTISET`, and `LIST` columns. Both are **opaque wrappers, not decomposed values**:
```python
row_val.raw # bytes the server's textual representation, e.g. b"ROW('Alice',30)"
row_val.schema # str the column's declared schema
row_val.raw # bytes: the server's textual representation, e.g. b"ROW('Alice',30)"
row_val.schema # str: the column's declared schema
coll_val.raw # bytes, e.g. b'LIST{10,20,30}'
coll_val.kind # 'set' | 'multiset' | 'list' | 'collection'
@ -110,7 +114,7 @@ coll_val.element_schema # str
They are not iterable and do not expose fields by name. Fully parsing a composite type means reimplementing JDBC's `IfxComplexInput`, which we haven't done.
If you need individual fields today, project them in SQL — the server does the decomposition for you and you get ordinary typed columns back:
If you need individual fields today, project them in SQL. The server does the decomposition for you and you get ordinary typed columns back:
```python
cur.execute("SELECT person.name, person.age FROM staff") # str, int

View File

@ -55,7 +55,7 @@ docker logs -f informix-dev
```
<Aside type="tip">
The `--privileged` flag is required by the dev image — it tries to manage shared memory limits. For production servers this isn't a thing.
The `--privileged` flag is required by the dev image, which tries to manage shared memory limits. For production servers this isn't a thing.
</Aside>
## 3. Run your first query
@ -88,7 +88,7 @@ Then:
python hello.py
```
You should see five rows from Informix's system catalog. If you do, congratulations you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
You should see five rows from Informix's system catalog. If you do, congratulations: you've spoken SQLI to an IBM database from pure Python with zero native code in the call stack.
## What just happened
@ -119,7 +119,12 @@ with informix_db.connect(host="127.0.0.1", port=9088, user="informix",
print(cur.fetchone())
```
`?` and `:1` both work — Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
`?` and `:1` both work. Informix's native paramstyle is `numeric`, but `?` is supported as a synonym.
Rows come back as plain tuples. If you would rather read them by column
name, pass `row_factory=informix_db.Row` to `connect()` and you get
`row["name"]` and `row.name` alongside `row[0]`. See
[the API reference](/reference/api/#rows).
## 5. Use the connection pool
@ -144,7 +149,7 @@ with pool.connection() as conn:
pool.close()
```
The pool is thread-safe and has a per-connection wire lock accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
The pool is thread-safe and has a per-connection wire lock, so accidental sharing across threads doesn't corrupt the wire stream, though PEP 249 advice still holds (one connection per thread).
## What's next
@ -154,7 +159,7 @@ The pool is thread-safe and has a per-connection wire lock — accidental sharin
2. **Connecting to production?** [Connect with TLS →](/how-to/tls/) covers TLS-listener configuration and bring-your-own-context patterns.
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/) — and the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
3. **Bulk-loading?** [Bulk inserts (executemany) →](/how-to/executemany/), including the 53× transaction-vs-autocommit gotcha you'll hit otherwise.
4. **Migrating from IfxPy?** [Migrate from IfxPy →](/how-to/migrate-from-ifxpy/) covers the API differences and the things IfxPy does that we don't (yet).

View File

@ -7,7 +7,7 @@ sidebar:
import { Aside } from '@astrojs/starlight/components';
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
[IfxPy](https://pypi.org/project/IfxPy/) is IBM's official Python driver, a C extension that wraps the OneDB Client SDK (CSDK), which itself wraps the same SQLI wire protocol `informix-driver` speaks directly. It's the reasonable comparison: same protocol, same server, same workload, different transport.
Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Informix Developer Edition Docker container on the same host. Methodology and reproduction steps live in [`tests/benchmarks/compare/`](https://git.supported.systems/warehack.ing/informix-db/src/branch/main/tests/benchmarks/compare) in the repo.
@ -33,9 +33,9 @@ Numbers below are **median + IQR over 10+ rounds**, all against the same IBM Inf
### Bulk inserts at scale
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms** `informix-driver` is 1.6× faster.
The clearest win is bulk insert throughput. `executemany(10_000_rows)` runs in **161 ms** vs IfxPy's **259 ms**, so `informix-driver` is 1.6× faster.
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row N RTTs at ~80 µs each adds up to the 100 ms gap.
The mechanism is pipelining. Phase 33 changed `executemany` to send all N BIND+EXECUTE PDUs back-to-back **before** draining any response. IfxPy's C-level `IfxPy.execute(stmt, tuple)` makes one round-trip per row, and N RTTs at ~80 µs each adds up to the 100 ms gap.
```python
# Both drivers
@ -44,8 +44,8 @@ cur.executemany(
rows, # list of 10_000 tuples
)
# informix-driver: 161 ms — 10k PDUs sent, then 10k responses drained
# IfxPy: 259 ms — 10k round-trips, each blocking on response
# informix-driver: 161 ms (10k PDUs sent, then 10k responses drained)
# IfxPy: 259 ms (10k round-trips, each blocking on response)
```
### Containerized deployment
@ -62,7 +62,7 @@ IfxPy's deployment surface is dramatically larger:
- 92 MB IBM OneDB Client tarball
- `setuptools < 58` build pin
- `LD_LIBRARY_PATH` configuration for four directories
- `libcrypt.so.1` (deprecated 2018 missing on Arch, Fedora 35+, RHEL 9)
- `libcrypt.so.1` (deprecated 2018, missing on Arch, Fedora 35+, RHEL 9)
- C compiler in the build image
For slim images, multi-stage builds, FaaS deployments, or anywhere build-toolchain-on-the-runtime is friction, `informix-driver` is the only reasonable option.
@ -88,13 +88,13 @@ async def main():
rows = await cur.fetchall()
```
IfxPy has no async support every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
IfxPy has no async support, so every call blocks the event loop. Using IfxPy from FastAPI requires `loop.run_in_executor()` boilerplate, and the thread pool isn't connection-aware so you give up the natural fairness of an async pool.
## When IfxPy wins
### Large analytical fetches
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 515% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock meaningful but not disqualifying.
For queries pulling 10k+ rows where per-row decode cost dominates, IfxPy is currently 515% faster. The C-level `fetch_tuple` decoder is ~1.1 µs/row; our Python `parse_tuple_payload` is ~2.0 µs/row after Phase 39 (down from ~2.7 before). At 100k rows the gap is ~80 ms wall-clock, which is meaningful but not disqualifying.
The gap is closing phase by phase:
@ -109,7 +109,7 @@ If you're running analytical reports that pull millions of rows in a single SELE
### Workloads built around CSDK extensions
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API — see [the migration guide](/how-to/migrate-from-ifxpy/).
If your existing code uses IBM-specific cursor extensions (`cursor.callproc` with named parameters, IBM's specific scrollable cursor semantics around `last`/`prior`/`relative`, `cursor.set_chunk_size` for fetch tuning), the migration to `informix-driver` is straightforward but not zero-cost. We support the core PEP 249 surface plus our own scrollable cursor API. See [the migration guide](/how-to/migrate-from-ifxpy/).
## Methodology
@ -117,7 +117,7 @@ Benchmarks are pytest-benchmark fixtures in `tests/benchmarks/compare/` against
Reported numbers are **median over 10+ rounds**, with IQR included. Why median over mean: the first round of any run includes JIT warmup, page-cache miss, and a TCP slow-start round-trip. The mean is contaminated by these one-shot costs in a way that misrepresents steady-state behavior. Median + IQR is what we report.
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise — a fair reading is "515% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
IfxPy's IQR on the 100k-row SELECT is ~21% (Docker→host loopback noise, plus the C extension's allocation patterns). Our IQR is ~0.2%. The headline 1.15× ratio at 100k rows is partly that noise. A fair reading is "515% slower than IfxPy on large fetches", and the lower bound may already be within measurement noise.
To reproduce:
@ -146,4 +146,4 @@ Use IfxPy when:
- You're running large analytical SELECTs and the 515% decode-side gap matters
- You're constrained to Python ≤ 3.11 anyway
For everything else the cost-benefit favors `pip install informix-driver`.
For everything else, the cost-benefit favors `pip install informix-driver`.

View File

@ -8,7 +8,7 @@ sidebar:
The existing tools were not my style.
Every Informix driver in any language `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, Perl `DBD::Informix` — wraps either IBM's C Client SDK or the JDBC JAR. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
Every Informix driver in any language wraps either IBM's C Client SDK or the JDBC JAR. That covers `IfxPy`, the legacy `informixdb`, ODBC bridges, JPype/JDBC, and Perl's `DBD::Informix`. To our knowledge `informix-driver` is the **first pure-socket Informix driver in any language**.
## The problem with IBM's C SDK
@ -19,19 +19,19 @@ The IBM Informix Client SDK (CSDK), now packaged as part of OneDB Client, is a 9
- Permissive `CFLAGS` for the C extension build
- Manual download of the 92 MB ODBC tarball
- Four `LD_LIBRARY_PATH` directories
- `libcrypt.so.1` — deprecated in 2018, missing on Arch, Fedora 35+, RHEL 9
- `libcrypt.so.1`, deprecated in 2018 and missing on Arch, Fedora 35+, RHEL 9
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db` the distribution name dodges PyPI's 2008-vintage `informixdb` package, the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
For containerized deployments, ETL pipelines, FastAPI services, or anywhere Python lives and IBM's C SDK is friction, the friction compounds. `informix-driver`'s install is `pip install informix-driver` (`import informix_db`; the distribution name dodges PyPI's 2008-vintage `informixdb` package, while the import name is what you'd expect). The wheel is ~50 KB. There are zero runtime dependencies.
## What it does
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly — the same protocol IBM's JDBC driver uses, the same protocol the CSDK speaks under the hood. No native code is in the thread of execution.
`informix-driver` opens a TCP socket to an Informix server's SQLI listener and speaks the wire protocol directly. It is the same protocol IBM's JDBC driver uses, and the same one the CSDK speaks under the hood. No native code is in the thread of execution.
The wire protocol was reverse-engineered through three sources:
1. **Decompiled IBM JDBC driver** (`com.informix.jdbc.IfxConnection` and friends), used as a clean-room reference for PDU shapes and protocol semantics.
2. **Annotated `socat` captures** of real client/server traffic against the IBM Informix Developer Edition Docker image.
3. **Differential testing** against `IfxPy` — every codec path is tested against the C driver's behavior on the same data.
3. **Differential testing** against `IfxPy`, so every codec path is checked against the C driver's behavior on the same data.
The result is a PEP 249 compliant driver with a sync API, an async API (FastAPI / asyncio compatible), a connection pool, TLS support, smart-LOB read/write, scrollable cursors, fast-path stored procedure invocation, and bulk-insert / bulk-fetch performance within ~1060% of the C driver depending on workload.
@ -76,12 +76,12 @@ Every finding from a system-wide failure-mode audit (data correctness, wire safe
**0 critical, 0 high, 0 medium audit findings remain.** Every architectural change went through a Margaret Hamilton-style review focused on silent-failure modes, recovery paths, and documented invariants. Each documented invariant is paired with either a runtime guard or a CI tripwire test.
400+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 326/326 on **all three** of 12.10.FC12W1DE, 14.10.FC7W1DE, and 15.0.1.0.3DE — `make test-matrix` runs the lot.
470+ tests across unit / integration / benchmark suites. The integration suite runs against the official IBM Informix Developer Edition Docker images and passes 472/472 on 15.0.1.0.3DE and 14.10.FC7W1DE, and 471/472 on 12.10.FC12W1DE (the single skip is a common table expression, which 12.10 predates). `make test-matrix` runs the lot.
That matrix exists because it turned out to be needed. A user reported corrupted result sets on Informix 12; the cause was three framing bugs that affected every version including the one we tested against, and they'd survived because no fixture used the affected types. Testing one server and inferring the rest is how that happens.
## Read next
- **[Install & first query →](/start/quickstart/)** five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
- **[Compared to IfxPy →](/start/vs-ifxpy/)** full head-to-head benchmarks, methodology, and reproduction.
- **[Architecture →](/explain/architecture/)** — how the layers stack: socket, framing, codec, resultset, cursor.
- **[Install & first query →](/start/quickstart/)**: five minutes from `pip install` to a real SELECT against a Docker-hosted Informix.
- **[Compared to IfxPy →](/start/vs-ifxpy/)**: full head-to-head benchmarks, methodology, and reproduction.
- **[Architecture →](/explain/architecture/)**: how the layers stack, from socket through framing, codec, resultset, and cursor.

View File

@ -243,7 +243,7 @@
color: var(--sl-color-gray-2);
}
/* Supported Systems "joint" badge appears below every page's footer */
/* Supported Systems "joint" badge, appears below every page's footer */
.ifx-ss-badge {
margin-top: 3.5rem;
padding: 0;
@ -337,7 +337,7 @@
* ============================================================ */
@media (max-width: 640px) {
/* Defensive guard against any descendant forcing horizontal scroll
/* Defensive guard against any descendant forcing horizontal scroll:
the wire-dump's white-space: pre content was overflowing the
hero column and propagating up to the page. overflow-x: hidden
on .ifx-hero contains it without affecting page-level scroll. */

View File

@ -1,7 +1,7 @@
/*
* informix-driver docs theme
* - Charcoal base (no purple gradients, ever)
* - Amber accent CRT-monitor nod, distinct from sibling sites' cyan
* - Amber accent, a CRT-monitor nod distinct from sibling sites' cyan
* - Inter for body, IBM Plex Mono for technical bytes
*/
@ -82,7 +82,7 @@
--sl-color-hairline-shade: rgba(120, 80, 12, 0.28);
}
/* Tighten heading rhythm Starlight defaults are a touch loose for technical docs */
/* Tighten heading rhythm; Starlight defaults are a touch loose for technical docs */
.sl-markdown-content h2 {
margin-top: 2.5rem;
border-top: 1px solid var(--sl-color-hairline-light);
@ -104,7 +104,7 @@
border-radius: 4px;
}
/* Tables: dense, technical, with amber column rules — for type-mapping & benchmark tables */
/* Tables: dense, technical, amber column rules, for type-mapping & benchmark tables */
.sl-markdown-content table {
border-collapse: collapse;
font-variant-numeric: tabular-nums;
@ -124,7 +124,7 @@
padding: 0.5rem 0.75rem;
}
/* Anchor links underline, no rainbow */
/* Anchor links: underline, no rainbow */
.sl-markdown-content a:not(.sl-anchor-link) {
text-decoration: underline;
text-decoration-color: var(--sl-color-accent);

View File

@ -1,6 +1,6 @@
[project]
name = "informix-driver"
version = "2026.08.31.1"
version = "2026.09.03"
description = "Pure-Python driver for IBM Informix IDS — speaks the SQLI wire protocol over raw sockets. No CSDK, no JVM, no native libraries."
readme = "README.md"
license = { text = "MIT" }

View File

@ -50,6 +50,7 @@ from .pool import (
PoolTimeoutError,
create_pool,
)
from .rows import Row
# PEP 249 module-level globals
apilevel = "2.0"
@ -85,6 +86,7 @@ __all__ = [
"PoolClosedError",
"PoolTimeoutError",
"ProgrammingError",
"Row",
"RowValue",
"ServerCapabilities",
"Warning",
@ -111,6 +113,7 @@ def connect(
client_locale: str = "en_US.8859-1",
env: dict[str, str] | None = None,
autocommit: bool = False,
row_factory: object | None = None,
tls: bool | ssl.SSLContext = False,
tls_server_hostname: str | None = None,
) -> Connection:
@ -153,4 +156,5 @@ def connect(
client_locale=client_locale,
env=env,
autocommit=autocommit,
row_factory=row_factory,
)

View File

@ -29,6 +29,29 @@ class ProtocolError(Exception):
"""Raised when wire bytes can't be parsed (truncated stream, bad framing)."""
# Every way a wire read/write can fail, as one tuple.
#
# This exists because the tuple was written out by hand in six places and
# drifted: four of them caught ``(ProtocolError, OSError)`` — which
# ``IfxSocket`` never raises. It converts *every* socket failure, including
# clean EOF, into ``OperationalError``, and that is a ``DatabaseError``, not
# an ``OSError``. So those four handlers could not catch the thing they
# existed to catch. The worst of them sat in a ``weakref.finalize`` callback,
# where the escaping exception is printed to stderr and swallowed, leaving a
# desynchronised connection to be returned to the pool marked healthy.
#
# Import this rather than re-typing the members. A tuple in one place cannot
# drift out of sync with itself.
def _wire_error_types() -> tuple[type[BaseException], ...]:
# Imported lazily: exceptions.py must stay free of protocol imports.
from .exceptions import InterfaceError, OperationalError
return (ProtocolError, OSError, OperationalError, InterfaceError)
WIRE_ERRORS: tuple[type[BaseException], ...] = _wire_error_types()
# ---------------------------------------------------------------------------
# Writer
# ---------------------------------------------------------------------------
@ -269,6 +292,13 @@ class BufferedSocketReader(IfxStreamReader):
return b
def skip(self, n: int) -> None:
# The base reader's skip delegates to read_exact, which returns
# b"" for n <= 0. This one advances the cursor arithmetically, so
# without the guard a negative n *rewinds* it — silently
# re-decoding bytes already consumed as if they were the next
# field. Same divergence as read_exact, which does guard.
if n <= 0:
return
sock = self._sock
sock.fill_recv_buf(n)
sock._recv_pos += n

View File

@ -25,7 +25,7 @@ from dataclasses import dataclass
from datetime import timedelta as _timedelta
from types import MappingProxyType
from ._protocol import IfxStreamReader
from ._protocol import IfxStreamReader, ProtocolError
from ._types import IfxType, base_type, is_nullable
from .converters import (
_DOUBLE_NULL,
@ -248,6 +248,114 @@ _COMPOSITE_UDT_TYPES = frozenset({
_NUMERIC_TYPES = frozenset({_TC_DECIMAL, _TC_MONEY})
def _read_udt_envelope(payload: bytes, offset: int) -> tuple[int, bytes | None]:
"""Read the UDT wire envelope: ``[1-byte indicator][int32 length][data]``.
Returns ``(new_offset, body)`` with ``body`` ``None`` when the
indicator says NULL. The length field is present either way it
belongs to the envelope, not the value so the offset advances by
5 for a NULL and 5 + length otherwise.
This exists because the same envelope was hand-written four times and
three of the copies were wrong, each in a different way:
* BOOLEAN read ``encoded_length`` (1 byte) instead of the envelope,
leaving 5 bytes behind.
* The composite-UDT branch returned on the indicator without reading
the length, leaving 4 bytes behind.
* BLOB/CLOB read a flat 72 bytes, leaving 77 behind.
Only the UDTVAR(lvarchar) copy was right, and only after two separate
fixes. Four copies of one three-line format is four chances to get it
wrong; this is one.
"""
indicator = payload[offset]
offset += 1
length = int.from_bytes(payload[offset : offset + 4], "big", signed=True)
offset += 4
if indicator == 1:
return offset, None
if length < 0:
raise ProtocolError(
f"UDT envelope declared a negative length ({length}) at payload "
f"offset {offset - 4}; the wire is desynchronised"
)
body = payload[offset : offset + length]
offset += length
return offset, body
def _decode_lob_locator(body: bytes, extended_id: int):
"""Turn a smart-LOB envelope body into a Blob/ClobLocator.
The body is the 72-byte locator **hex-encoded as 144 ASCII characters**,
not the raw locator. We previously consumed a flat 72 bytes, which took
the first half of that hex text and handed it to ``BlobLocator`` as if
it were binary so ``BlobLocator.raw`` never actually held a locator.
It went unnoticed because ``read_blob_column`` resolves LOBs through
the server-side ``lotofile`` function and never uses the locator.
"""
cls = BlobLocator if extended_id == 10 else ClobLocator
try:
raw = bytes.fromhex(body.decode("ascii"))
except (ValueError, UnicodeDecodeError):
# Not hex — surface what arrived rather than guessing, but only if
# it is already the right size for a locator.
raw = bytes(body)
if len(raw) != 72:
raise ProtocolError(
f"smart-LOB locator decoded to {len(raw)} bytes, expected 72 "
f"(envelope body was {len(body)} bytes)"
)
return cls(raw=raw)
def _row_not_consumed(
offset: int, payload_len: int, columns: list[ColumnInfo]
) -> None:
"""Raise when a decoded row didn't consume exactly its payload.
THIS IS THE CHECK THAT WOULD HAVE CAUGHT EVERY FRAMING BUG WE SHIPPED.
Thirteen of them reached users, and all thirteen were the same defect:
a column consumed the wrong number of bytes. Not one raised at the
point of the mistake the wrong width produced a plausible value and
corrupted whatever came *next*, so the damage surfaced in a different
column, a different row, or a different statement.
Every one was detectable here for free. ``payload`` is a fully
extracted ``bytes`` object of known length (the wire's even-alignment
pad is consumed separately by the caller), so after decoding N columns
the offset must land exactly on the end. Anything else means the codec
and the server disagree about the row's shape, and continuing can only
produce wrong answers.
A previous attempt at this check was reverted, and the comment
explaining why survived long after its reason did: it cited the
LVARCHAR odd-length pad, which was itself a bug and has since been
removed. With that gone the invariant is exact.
The message names the column shape because "row decode failed" is not
actionable the whole point is to say *which* column shape and by how
many bytes, so the next report arrives with the answer in it.
"""
delta = offset - payload_len
direction = "over-read" if delta > 0 else "under-read"
shape = ", ".join(
f"{c.name}:tc={c.type_code}"
f"{'/ext=' + c.extended_name if c.extended_name else ''}"
f"/enclen={c.encoded_length}"
for c in columns
)
raise ProtocolError(
f"row decoder {direction} by {abs(delta)} byte(s): consumed "
f"{offset} of {payload_len} payload bytes. This means the driver "
f"and the server disagree about a column's wire width, so every "
f"column after the offending one is unreliable. Column shape: "
f"[{shape}]"
)
def _packed_width(encoded_length: int) -> int:
"""On-wire byte width for the four types whose ``encoded_length``
packs two fields into ``(high << 8) | low``: DECIMAL, MONEY,
@ -540,6 +648,12 @@ def compile_row_decoder(
# Unknown kind — abort codegen, caller falls back.
return None
# Same end-of-row reconciliation the interpreted paths do. The
# generated function is the hot path, so this must be emitted here
# too — a check that only guards the slow path guards nothing in
# production.
lines.append(" if offset != len(payload):")
lines.append(" _row_short(offset, len(payload))")
if val_names:
lines.append(f" return ({', '.join(val_names)},)")
else:
@ -564,6 +678,13 @@ def compile_row_decoder(
"_decode_datetime": _decode_datetime,
"_decode_interval": _decode_interval,
"_legacy_dispatch_one_column": _legacy_dispatch_one_column,
# Bound to this shape's columns so the raised message can name them
# without the generated source having to carry the list itself.
"_row_short": (
lambda offset, payload_len, _cols=columns: _row_not_consumed(
offset, payload_len, _cols
)
),
"_UNPACK_SHORT": _UNPACK_SHORT,
"_UNPACK_INT": _UNPACK_INT,
"_UNPACK_LONG": _UNPACK_LONG,
@ -610,11 +731,16 @@ def _legacy_dispatch_one_column(
"""
# BLOB / CLOB locator (UDTFIXED + extended_id 10/11)
if tc == _TC_UDTFIXED and col.extended_id in (10, 11):
width = col.encoded_length
raw = payload[offset:offset + width]
offset += width
cls = BlobLocator if col.extended_id == 10 else ClobLocator
return offset, cls(raw=bytes(raw))
# Smart LOBs use the UDT envelope, NOT a flat encoded_length field.
# Measured on Informix 15: a populated BLOB column is 149 bytes
# ([ind=0][len=144][144 hex chars]) and a NULL one is 5, while
# encoded_length reports 72. Consuming 72 left 77 bytes behind and
# corrupted every column after a LOB. Found by the end-of-row
# reconciliation check, which fired on its first run.
offset, body = _read_udt_envelope(payload, offset)
if body is None:
return offset, None
return offset, _decode_lob_locator(body, col.extended_id)
# BOOLEAN. The server describes it as UDTFIXED (41) with
# extended_name='boolean' and encoded_length=1, but ``encoded_length``
@ -628,24 +754,28 @@ def _legacy_dispatch_one_column(
if tc == _TC_UDTFIXED and (
col.extended_name == "boolean" or col.extended_id == 5
):
indicator = payload[offset]
offset += 1
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
offset += 4
raw = payload[offset:offset + length]
offset += length
if indicator == 1:
offset, body = _read_udt_envelope(payload, offset)
if body is None:
return offset, None
return offset, bool(raw and raw[0] in (ord("t"), ord("T"), 1))
return offset, bool(body and body[0] in (ord("t"), ord("T"), 1))
# ROW / COLLECTION composite UDT
if tc in _COMPOSITE_UDT_TYPES:
indicator = payload[offset]
offset += 1
if indicator == 1:
return offset, None
# The 4-byte length is part of the envelope and is present even when
# the indicator says NULL — identical to the UDTVAR(lvarchar) branch
# below, which decodes the same `[ind][int32 len][data]` shape.
# Returning early on the indicator left those 4 bytes unread and
# corrupted the following column. Wire evidence, Informix 15,
# (INT, SET(INT), VARCHAR) with a NULL set:
# 00 00 00 08 | 01 | 00 00 00 00 | 03 78 79 7a
# a = 8 | ind | length = 0 | [3]"xyz"
# Before the fix the VARCHAR decoded as '' instead of 'xyz'.
length = int.from_bytes(payload[offset:offset + 4], "big", signed=True)
offset += 4
if indicator == 1:
return offset, None
raw = bytes(payload[offset:offset + length])
offset += length
if tc == _TC_ROW:
@ -830,6 +960,8 @@ def parse_tuple_payload(
payload, offset, tc, col, encoding
)
values.append(value)
if offset != len(payload):
_row_not_consumed(offset, len(payload), columns)
return tuple(values)
# Legacy slow path (no pre-compiled readers).
@ -936,11 +1068,13 @@ def parse_tuple_payload(
# we read here are an opaque server-side reference, NOT the
# actual data. Phase 10 lets users fetch via lotofile + SQ_FILE.
if tc == _TC_UDTFIXED and col.extended_id in (10, 11):
width = col.encoded_length
raw = payload[offset:offset + width]
offset += width
cls = BlobLocator if col.extended_id == 10 else ClobLocator
values.append(cls(raw=bytes(raw)))
# UDT envelope, not a flat encoded_length field — see the
# matching branch in _legacy_dispatch_one_column.
offset, body = _read_udt_envelope(payload, offset)
values.append(
None if body is None
else _decode_lob_locator(body, col.extended_id)
)
continue
# BOOLEAN — UDT envelope, not a bare byte. See the matching branch
@ -948,18 +1082,11 @@ def parse_tuple_payload(
if tc == _TC_UDTFIXED and (
col.extended_name == "boolean" or col.extended_id == 5
):
indicator = payload[offset]
offset += 1
length = int.from_bytes(
payload[offset:offset + 4], "big", signed=True
offset, body = _read_udt_envelope(payload, offset)
values.append(
None if body is None
else bool(body and body[0] in (ord("t"), ord("T"), 1))
)
offset += 4
raw = payload[offset:offset + length]
offset += length
if indicator == 1:
values.append(None)
else:
values.append(bool(raw and raw[0] in (ord("t"), ord("T"), 1)))
continue
# ROW / COLLECTION (Phase 12): composite UDTs. Wire format is
@ -976,13 +1103,15 @@ def parse_tuple_payload(
if tc in _COMPOSITE_UDT_TYPES:
indicator = payload[offset]
offset += 1
if indicator == 1: # null
values.append(None)
continue
# Length is present even when NULL — see the matching branch in
# _legacy_dispatch_one_column for the wire evidence.
length = int.from_bytes(
payload[offset:offset + 4], "big", signed=True
)
offset += 4
if indicator == 1: # null
values.append(None)
continue
raw = bytes(payload[offset:offset + length])
offset += length
if tc == _TC_ROW:
@ -1048,4 +1177,6 @@ def parse_tuple_payload(
# by Python's slicing semantics for strings — short = harmless).
# If a future protocol message produces actual garbage here, add a
# branch-local check at the offending dispatch path.
if offset != len(payload):
_row_not_consumed(offset, len(payload), columns)
return tuple(values)

View File

@ -17,11 +17,35 @@ the rest of the protocol layer.
from __future__ import annotations
import contextlib
import os
import socket
import ssl
from ._protocol import ProtocolError
from .exceptions import InterfaceError, OperationalError
def _max_read_bytes() -> int:
"""Ceiling for a single length-prefixed read, from IFX_MAX_READ_BYTES.
256 MiB by default: comfortably above any row, string table, or blob
chunk a real query produces, and far below the values a desynced
stream invents. A genuinely larger single value is possible (a very
large TEXT column read in one go), which is why the knob exists.
"""
raw = os.environ.get("IFX_MAX_READ_BYTES")
if raw:
try:
value = int(raw)
except ValueError:
value = 0
if value > 0:
return value
return 256 * 1024 * 1024
MAX_READ_BYTES = _max_read_bytes()
# A ``tls`` parameter to ``IfxSocket`` accepts:
# False (default) — plain TCP
# True — TLS with verification disabled (dev / self-signed)
@ -124,10 +148,42 @@ class IfxSocket:
raise OperationalError(f"write failed: {e}") from e
def read_exact(self, n: int) -> bytes:
"""Read exactly ``n`` bytes or raise on EOF / timeout."""
"""Read exactly ``n`` bytes or raise on EOF / timeout.
Consumes the read-ahead buffer before touching the socket. This
matters because two readers share one stream: ``BufferedSocketReader``
fills ``_recv_buf`` (over-reading by design, up to ``_recv_size``),
while ``Connection._drain_to_eot``, ``_raise_sq_err`` and the
login path call this method directly. Recv'ing here while bytes
sat unconsumed in the buffer would skip them, and skipped bytes
in a length-framed protocol don't announce themselves — the next
read lands mid-field and every subsequent one is wrong.
No workload triggers it today: the server sends one response per
request, so recv returns exactly that response and the buffered
reader consumes all of it before control returns here. That is a
property of the traffic, not of the code. Pipelined executemany
already puts multiple responses in flight, and the buffer is
connection-scoped precisely so read-ahead can cross response
boundaries. Making the two paths agree by construction costs one
branch on a cold path.
"""
if self._sock is None:
raise InterfaceError("socket is closed")
if n <= 0:
return b""
wanted = n
chunks: list[bytes] = []
buffered = len(self._recv_buf) - self._recv_pos
if buffered > 0:
take = min(buffered, n)
chunks.append(
bytes(self._recv_buf[self._recv_pos : self._recv_pos + take])
)
self._recv_pos += take
n -= take
if n == 0:
return chunks[0]
remaining = n
while remaining > 0:
try:
@ -138,7 +194,8 @@ class IfxSocket:
if not chunk:
self._force_close()
raise OperationalError(
f"server closed connection mid-read (wanted {n} bytes, got {n - remaining})"
f"server closed connection mid-read "
f"(wanted {wanted} bytes, got {wanted - remaining})"
)
chunks.append(chunk)
remaining -= len(chunk)
@ -155,6 +212,21 @@ class IfxSocket:
"""
if self._sock is None:
raise InterfaceError("socket is closed")
if need > MAX_READ_BYTES:
# ``need`` is almost always a length field straight off the
# wire. Trusting it means a corrupt or desynced stream turns
# into an allocation of whatever that field happened to say —
# a garbage 0x7FFFFFFF reads as a 2 GB request, and the loop
# below sits in recv until the read timeout while the buffer
# grows. Refusing names the number instead, which is the
# diagnostic: a length that absurd means framing is already
# lost upstream, not that the row is genuinely that big.
raise ProtocolError(
f"refusing to read {need} bytes in one field (limit "
f"{MAX_READ_BYTES}). Either the wire has desynced, or a "
f"single value really is this large — raise the limit "
f"with the IFX_MAX_READ_BYTES environment variable."
)
avail = len(self._recv_buf) - self._recv_pos
if avail >= need:
return

View File

@ -62,7 +62,10 @@ from __future__ import annotations
import asyncio
import contextlib
import functools
import threading
import weakref
from collections.abc import AsyncIterator, Awaitable, Callable
from concurrent.futures import ThreadPoolExecutor
from typing import Any, TypeVar
from . import connect as _sync_connect
@ -79,6 +82,61 @@ def _to_thread(fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
return asyncio.to_thread(fn, *args, **kwargs)
def _run_on(
executor: ThreadPoolExecutor,
fn: Callable[..., T],
*args: Any,
**kwargs: Any,
) -> Awaitable[T]:
"""Await ``fn`` on a specific executor rather than the loop's default."""
loop = asyncio.get_running_loop()
return loop.run_in_executor(executor, functools.partial(fn, *args, **kwargs))
def _connection_executor(conn: _SyncConnection) -> ThreadPoolExecutor:
"""The dedicated worker thread for one connection, created on demand.
``asyncio.to_thread`` runs on the event loop's default executor, which
the whole process shares and which is sized from the CPU count
``min(32, cpu_count + 4)``, so six threads on a two-CPU container.
Two consequences, both silent.
A cancelled await does not stop its worker. ``asyncio.to_thread``
cannot interrupt the thread, so it keeps running the wire call until
the read timeout expires. Cancellation is ordinary in a web app a
client disconnect cancels the request task so a handful of them
pins every thread in the shared pool and unrelated ``to_thread`` work
anywhere else in the process stops dead. Measured: six cancelled
calls against a six-worker default executor starve an unrelated
``to_thread`` indefinitely.
And pool concurrency was capped by the same number without saying so.
A pool with ``max_size=20`` on a two-CPU box ran six queries at once.
One thread per connection is the right size rather than a compromise:
the sync connection serializes every wire operation on its own lock,
so a second thread could do nothing but wait for the first. It also
rules out a deadlock that a shared pool-sized executor invites with
N threads and N connections, N tasks blocked in ``acquire`` occupy
every thread while the connection they are waiting for is held by a
task that now needs a thread of its own to finish and release it.
The executor lives on the sync connection so it survives being
returned to the pool and handed out again. The finalizer is the
backstop for a connection dropped without ``close()``: a
``ThreadPoolExecutor`` that is never shut down leaves its worker
parked on the work queue for the life of the process.
"""
executor = getattr(conn, "_async_executor", None)
if executor is None:
executor = ThreadPoolExecutor(
max_workers=1, thread_name_prefix="informix-conn"
)
conn._async_executor = executor
weakref.finalize(conn, executor.shutdown, wait=False)
return executor
class AsyncCursor:
"""Async wrapper over a sync :class:`Cursor`. Each I/O call awaits
a thread-offloaded version of the sync operation.
@ -88,10 +146,14 @@ class AsyncCursor:
paying the thread-hop cost.
"""
__slots__ = ("_cur",)
__slots__ = ("_cur", "_run")
def __init__(self, cur: _SyncCursor):
def __init__(self, cur: _SyncCursor, run: Callable[..., Awaitable[Any]]):
self._cur = cur
# The owning connection's runner, so cursor I/O lands on that
# connection's dedicated thread rather than the shared default
# executor. See _connection_executor.
self._run = run
# -- Pass-through synchronous attributes (no I/O) ---------------------
@ -120,32 +182,32 @@ class AsyncCursor:
async def execute(
self, operation: str, parameters: Any = None
) -> None:
await _to_thread(self._cur.execute, operation, parameters)
await self._run(self._cur.execute, operation, parameters)
async def executemany(
self, operation: str, seq_of_parameters: Any
) -> None:
await _to_thread(
await self._run(
self._cur.executemany, operation, list(seq_of_parameters)
)
async def fetchone(self) -> tuple | None:
return await _to_thread(self._cur.fetchone)
return await self._run(self._cur.fetchone)
async def fetchmany(self, size: int | None = None) -> list[tuple]:
return await _to_thread(self._cur.fetchmany, size)
return await self._run(self._cur.fetchmany, size)
async def fetchall(self) -> list[tuple]:
return await _to_thread(self._cur.fetchall)
return await self._run(self._cur.fetchall)
async def close(self) -> None:
await _to_thread(self._cur.close)
await self._run(self._cur.close)
# Phase 10/11 BLOB helpers (preserve the sync API surface)
async def read_blob_column(
self, sql: str, params: tuple = ()
) -> bytes | None:
return await _to_thread(self._cur.read_blob_column, sql, params)
return await self._run(self._cur.read_blob_column, sql, params)
async def write_blob_column(
self,
@ -155,7 +217,7 @@ class AsyncCursor:
*,
clob: bool = False,
) -> None:
await _to_thread(
await self._run(
functools.partial(
self._cur.write_blob_column,
sql, blob_data, params, clob=clob,
@ -177,14 +239,23 @@ class AsyncCursor:
class AsyncConnection:
"""Async wrapper over a sync :class:`Connection`."""
__slots__ = ("_conn",)
__slots__ = ("_conn", "_executor")
def __init__(self, conn: _SyncConnection):
self._conn = conn
self._executor = _connection_executor(conn)
def _run(self, fn: Callable[..., T], *args: Any, **kwargs: Any) -> Awaitable[T]:
"""Run a blocking connection call on this connection's own thread."""
return _run_on(self._executor, fn, *args, **kwargs)
@classmethod
async def connect(cls, *args: Any, **kwargs: Any) -> AsyncConnection:
"""Open a connection. Same parameters as :func:`informix_db.connect`."""
"""Open a connection. Same parameters as :func:`informix_db.connect`.
The connect itself still goes to the default executor -- there is
no connection yet to own a thread, and it is one bounded call.
"""
sync_conn = await _to_thread(
functools.partial(_sync_connect, *args, **kwargs)
)
@ -195,22 +266,46 @@ class AsyncConnection:
return self._conn.closed
async def cursor(self) -> AsyncCursor:
sync_cur = await _to_thread(self._conn.cursor)
return AsyncCursor(sync_cur)
sync_cur = await self._run(self._conn.cursor)
return AsyncCursor(sync_cur, self._run)
async def commit(self) -> None:
await _to_thread(self._conn.commit)
await self._run(self._conn.commit)
async def rollback(self) -> None:
await _to_thread(self._conn.rollback)
await self._run(self._conn.rollback)
async def close(self) -> None:
await _to_thread(self._conn.close)
"""Close the connection and stop the thread that served it.
One thread per connection is only affordable if the thread goes
away with the connection. ``ThreadPoolExecutor`` workers park on
the work queue rather than exiting when idle, so an executor that
is never shut down leaks its thread for the life of the process
the ``weakref.finalize`` in ``_connection_executor`` is a backstop
for connections dropped without ``close()``, not a substitute for
closing here.
``wait=False`` because we are on the event loop: the worker has
just finished the close and needs no waiting, and blocking the
loop to confirm that would be the one thing this module exists to
avoid.
"""
try:
await self._run(self._conn.close)
finally:
self._executor.shutdown(wait=False)
# Drop the reference too. A shut-down executor rejects new
# work, so leaving it attached would turn any later wrap of
# this sync connection into a RuntimeError rather than a
# fresh thread.
with contextlib.suppress(AttributeError):
del self._conn._async_executor
async def fast_path_call(
self, signature: str, *params: object
) -> list[object]:
return await _to_thread(self._conn.fast_path_call, signature, *params)
return await self._run(self._conn.fast_path_call, signature, *params)
# Async context-manager support
async def __aenter__(self) -> AsyncConnection:
@ -249,14 +344,69 @@ class AsyncConnectionPool:
return self._pool.idle_count
async def acquire(self, timeout: float | None = None) -> AsyncConnection:
sync_conn = await _to_thread(self._pool.acquire, timeout)
"""Acquire a connection, surviving cancellation of the waiter.
``asyncio.to_thread`` cannot interrupt its worker. If the task
awaiting an acquire is cancelled while the worker is still
blocked waiting for a free connection, the worker eventually
succeeds and hands back a connection **that nobody owns** it
is checked out of the pool and never returned. Repeat that and
the pool starves to death.
This is not hypothetical for anyone serving HTTP: a client
disconnecting cancels the request task, and under load those
cancellations land precisely while waiting for a connection. The
pool then dies one slot at a time, and only under load.
So the inner future is shielded cancelling the caller must not
orphan a result we still need to see and if the caller does go
away, a callback returns whatever the worker produced to the
pool. ``add_done_callback`` fires immediately when the future has
already resolved, so the "worker finished a moment before the
cancellation" race is covered by the same code path.
"""
fut = asyncio.ensure_future(_to_thread(self._pool.acquire, timeout))
try:
sync_conn = await asyncio.shield(fut)
except asyncio.CancelledError:
fut.add_done_callback(self._return_orphan)
raise
return AsyncConnection(sync_conn)
def _return_orphan(self, fut: asyncio.Future) -> None:
"""Give back a connection whose acquirer was cancelled.
Runs on the event loop, so it must not block: ``release`` takes
the connection's wire lock and can wait. Hand off to a short-lived
daemon thread rather than the running loop this path also has to
work while the loop is shutting down, which is exactly when
``create_task`` is unavailable.
"""
if fut.cancelled() or fut.exception() is not None:
return
conn = fut.result()
def _release() -> None:
# broken=False: the connection was never handed to anyone, so
# its wire is untouched and it is safe to reuse. Evicting here
# would trade a leak for needless reconnect churn.
with contextlib.suppress(Exception):
self._pool.release(conn, broken=False)
threading.Thread(target=_release, daemon=True).start()
async def release(
self, conn: AsyncConnection, *, broken: bool = False
) -> None:
await _to_thread(
functools.partial(self._pool.release, conn._conn, broken=broken)
# On the connection's own thread, not the default executor. That
# thread is idle by definition — the caller is done with the
# connection — and it keeps the release off a shared pool that
# tasks blocked in ``acquire`` may have filled. Release has to
# win that race: it is what frees the connection they are
# waiting for.
await _run_on(
conn._executor,
functools.partial(self._pool.release, conn._conn, broken=broken),
)
@contextlib.asynccontextmanager

View File

@ -18,6 +18,7 @@ import socket as socket_mod
import ssl
import struct
import threading
import weakref
from io import BytesIO
from pathlib import Path
@ -38,10 +39,16 @@ from ._messages import (
SLHeader,
StmtOptions,
)
from ._protocol import IfxStreamReader, IfxStreamWriter, ProtocolError, make_pdu_writer
from ._protocol import (
WIRE_ERRORS,
IfxStreamReader,
IfxStreamWriter,
ProtocolError,
make_pdu_writer,
)
from ._socket import IfxSocket
from .cursors import Cursor
from .exceptions import InterfaceError, OperationalError
from .exceptions import InterfaceError, OperationalError, ProgrammingError
# Default capability bits the JDBC reference sends. Validated against
# 01-connect-only.socat.log via the PDU diff in tests/test_pdu_match.py:
@ -199,6 +206,62 @@ _DEFAULT_ENV: dict[str, str] = {
}
class _WireLock:
"""A reentrant lock that can say whether *this* thread already holds it.
``threading.RLock`` cannot answer that question in public API, and
the difference is not academic. The cursor finalizer runs at GC time
on whatever thread happened to allocate including a thread that is
at that moment mid-statement holding this lock. It probes with
``acquire(blocking=False)`` intending to mean "is anyone using the
wire?", but an RLock grants a reentrant acquire to its own owner, so
the probe returns True and the finalizer sends CLOSE/RELEASE into the
middle of the statement it interrupted. The victim gets ``-208``.
Reachability is not exotic. Refcounting frees a dropped cursor
immediately, before the next statement, which is why this went
unnoticed but a cursor caught in a reference cycle waits for a
collection instead, and cycles are ordinary in Python. Any traceback
that holds a cursor makes one.
``held_by_current_thread`` reads ``_depth``/``_owner`` without the
lock, which is safe: both are mutated only under it, and the only
values another thread can leave behind are a depth of zero or an
owner that isn't us. Either way the answer is False, correctly.
"""
__slots__ = ("_depth", "_lock", "_owner")
def __init__(self) -> None:
self._lock = threading.RLock()
self._owner: int | None = None
self._depth = 0
def acquire(self, blocking: bool = True, timeout: float = -1) -> bool:
acquired = self._lock.acquire(blocking, timeout)
if acquired:
self._owner = threading.get_ident()
self._depth += 1
return acquired
def release(self) -> None:
self._depth -= 1
if self._depth == 0:
self._owner = None
self._lock.release()
def __enter__(self) -> _WireLock:
self.acquire()
return self
def __exit__(self, *exc_info: object) -> None:
self.release()
@property
def held_by_current_thread(self) -> bool:
return self._depth > 0 and self._owner == threading.get_ident()
class Connection:
"""A SQLI session. Owns one TCP socket and the post-login state.
@ -220,6 +283,7 @@ class Connection:
client_locale: str = "en_US.8859-1",
env: dict[str, str] | None = None,
autocommit: bool = False, # honored from Phase 3 onward
row_factory: object | None = None,
tls: bool | ssl.SSLContext = False,
tls_server_hostname: str | None = None,
):
@ -243,7 +307,7 @@ class Connection:
# this lock with a timeout, then calls ``conn.rollback()`` —
# which itself acquires the lock. Same thread, two acquires.
# Reentrance must be cheap and correct.
self._wire_lock = threading.RLock()
self._wire_lock = _WireLock()
# Phase 29: deferred-cleanup queue for cursor finalizers that
# couldn't acquire the wire lock at GC time. Each entry is a
# PDU's worth of bytes (typically a CLOSE or RELEASE) that
@ -263,12 +327,21 @@ class Connection:
# under ``_wire_lock``.
self._pending_cleanup: list[bytes] = []
self._cleanup_lock = threading.Lock()
# Weak ref to the scrollable cursor currently holding the
# session's statement slot, or None. Weak so that abandoning a
# scrollable cursor still lets its finalizer run — a strong ref
# here would keep the very object alive whose GC we depend on.
self._open_scroll_cursor: weakref.ref | None = None
# Logged-DB transaction state: True iff there's an open server-side
# transaction (SQ_BEGIN sent, not yet committed/rolled-back). The
# cursor uses this to decide whether to send an implicit SQ_BEGIN
# before the next DML in non-autocommit mode. We default to "no
# open txn" — the first DML will trigger SQ_BEGIN.
self._in_transaction = False
# Default row type for cursors from this connection. None means
# plain tuples, which is the zero-cost default; see
# informix_db.rows for the opt-in named-access type.
self.row_factory = row_factory
# Tri-state: True after first successful SQ_BEGIN, False after
# an unlogged-DB rejection (-201). None until we've tried.
# Used to avoid repeatedly probing on unlogged DBs.
@ -357,22 +430,80 @@ class Connection:
raise InterfaceError("connection is closed")
return Cursor(self, scrollable=scrollable)
def _send_pdu(self, pdu: bytes) -> None:
def _send_pdu(self, pdu: bytes, *, statement_boundary: bool = False) -> None:
"""Send an assembled PDU. Used by Cursor.
Phase 29: opportunistically drains any pending cleanup PDUs
from the deferred-cleanup queue *before* sending the new PDU.
Caller must hold ``_wire_lock`` (every actual call site already
does execute/executemany/_sfetch_at, commit, rollback,
fast_path_call, etc.). The drain happens under that lock so
the queued cleanup atomically completes before the next op.
Caller must hold ``_wire_lock`` (every actual call site does).
``statement_boundary=True`` additionally drains the deferred
cleanup queue first. **Only pass it when this PDU is the first
of a new statement**, meaning no statement is currently open
server-side.
The queue previously drained before *every* PDU, which was wrong
in a way that took a repro to see. ``SQ_CLOSE`` and
``SQ_RELEASE`` carry no statement identifier they act on the
server's *current* statement. A finalizer enqueues precisely
because it lost the race for the wire lock, which means another
thread is mid-statement; that thread's next ``_send_pdu`` was
then its own ``CURNAME``/``NFETCH``, and the drain released the
statement out from under it. The caller saw a nonsense error on
valid SQL: ``-208`` when injected before the first fetch,
``-267`` "transaction has been rolled back" between fetch
batches. Both point nowhere near the actual cause, and the
window is exactly the window in which enqueueing happens.
Draining only at a boundary costs nothing: cleanup that misses
one statement is picked up by the next.
"""
if self._closed:
raise InterfaceError("connection is closed")
if self._pending_cleanup:
if statement_boundary and self._pending_cleanup:
self._drain_pending_cleanup()
self._sock.write_all(pdu)
def _check_scroll_cursor_conflict(self, requester: object) -> None:
"""Refuse to start a statement while a scrollable cursor is open.
A server-side scrollable cursor occupies the session's statement
slot, and SQLI gives us no way to address around it: ``SQ_CLOSE``,
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on the session's current
statement.
The server does not refuse politely. Starting another statement
returns ``-285``, and the scrollable cursor is collateral damage
its next fetch comes back ``-267`` "the transaction has been
rolled back, all locks released". Two unattributable failures
from code that looks entirely ordinary: iterate a large result
set with a scrollable cursor, run a lookup query partway through.
Multiplexing is presumably expressible JDBC prefixes every
statement-scoped PDU with the statement id, which the server does
assign distinctly (0 and 1 for two concurrent cursors). But the
id alone is not sufficient: addressing the second cursor's
``SQ_SFETCH`` to its own id returns ``-259`` "cursor not open".
Until that is understood, refusing is the honest behaviour. It
costs the caller a second connection and it never corrupts.
"""
ref = self._open_scroll_cursor
if ref is None:
return
other = ref()
if (
other is None
or other is requester
or not getattr(other, "_server_cursor_open", False)
):
self._open_scroll_cursor = None
return
raise ProgrammingError(
"a scrollable cursor is open on this connection; Informix "
"allows only one statement per session, so running another "
"statement here would fail with -285 and destroy the "
"scrollable cursor as well. Close the scrollable cursor "
"first, or use a separate connection for the other statement."
)
def _enqueue_cleanup(self, pdus: list[bytes]) -> None:
"""Append cleanup PDUs to the deferred queue.
@ -402,7 +533,6 @@ class Connection:
discarded; the server-side resources they would have released
are freed when the session ends anyway.
"""
from ._protocol import ProtocolError
with self._cleanup_lock:
if not self._pending_cleanup:
@ -413,11 +543,42 @@ class Connection:
try:
self._sock.write_all(pdu)
self._drain_to_eot()
except (ProtocolError, OSError, OperationalError):
# Wire is unrecoverable; force-close. Subsequent
# ``_send_pdu`` will raise InterfaceError. Server
# cleanup of the remaining queued entries happens
# implicitly at session end.
except Exception as exc:
if getattr(exc, "sqlcode", None) is not None:
# The *server* rejected the cleanup — a stale entry
# for a cursor it no longer has. Queued cleanup goes
# stale routinely: the finalizer enqueues, then the
# cursor gets closed properly before the drain runs.
#
# This is not a wire problem. ``_raise_sq_err``
# self-drains the trailing SQ_EOT, so the wire is
# still aligned and the remaining entries are still
# worth sending.
#
# It must not escape, and it must not be treated as
# fatal. Both were wrong before: a stale CLOSE draws
# ``-267``, which is an OperationalError, which is in
# WIRE_ERRORS — so a stale queue entry force-closed a
# perfectly healthy connection. And this runs at the
# start of somebody else's statement, so letting it
# out would fail their good SQL with an error about a
# cursor they never opened.
_log.debug(
"deferred cleanup rejected by server (stale entry), "
"continuing: %r",
exc,
)
continue
if not isinstance(exc, WIRE_ERRORS):
_log.warning(
"unexpected error draining deferred cleanup: %r", exc
)
# No sqlcode means the failure came from the wire, not
# the server: the socket died, or framing desynced and we
# can no longer say where a response ends. Force-close.
# Subsequent ``_send_pdu`` raises InterfaceError. The
# server-side resources the remaining entries would have
# freed are released when the session ends anyway.
self._closed = True
with contextlib.suppress(Exception):
self._sock.close()
@ -596,7 +757,7 @@ class Connection:
# method but stable across versions; cheap (~50ns) and only
# checks the current thread. If it ever changes shape, drop
# this assert — the doc still names the precondition.
assert self._wire_lock._is_owned(), (
assert self._wire_lock.held_by_current_thread, (
"_ensure_transaction called without _wire_lock held; "
"the cursor method that called it must wrap its body in "
"`with self._conn._wire_lock:`"
@ -835,7 +996,6 @@ class Connection:
[short near_token_len][bytes name][optional pad][short SQ_EOT]
"""
from . import _errcodes
from ._protocol import ProtocolError
sqlcode = struct.unpack("!h", self._sock.read_exact(2))[0]
isamcode = struct.unpack("!h", self._sock.read_exact(2))[0]
@ -851,7 +1011,7 @@ class Connection:
if name_len & 1:
self._sock.read_exact(1)
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
except (ProtocolError, OSError):
except WIRE_ERRORS:
pass
# Phase 28: drain failure means wire desync — force-close so
# subsequent operations don't inherit the broken state.
@ -865,7 +1025,7 @@ class Connection:
next_tag = struct.unpack("!h", self._sock.read_exact(2))[0]
if next_tag == MessageType.SQ_EOT:
break
except (ProtocolError, OSError, OperationalError):
except WIRE_ERRORS:
self._closed = True
with contextlib.suppress(Exception):
self._sock.close()

View File

@ -30,7 +30,12 @@ from typing import TYPE_CHECKING, Any
from . import _errcodes
from ._messages import MessageType
from ._protocol import BufferedSocketReader, IfxStreamReader, make_pdu_writer
from ._protocol import (
WIRE_ERRORS,
BufferedSocketReader,
IfxStreamReader,
make_pdu_writer,
)
from ._resultset import (
ColumnInfo,
compile_column_readers,
@ -45,6 +50,7 @@ from .exceptions import (
NotSupportedError,
ProgrammingError,
)
from .rows import Row, make_row_class
if TYPE_CHECKING:
from .connections import Connection
@ -55,7 +61,6 @@ if TYPE_CHECKING:
_cursor_counter = itertools.count(1)
_NUMERIC_PLACEHOLDER_RE = __import__("re").compile(r":(\d+)")
# Phase 28: pre-built CLOSE and RELEASE PDU bytes for cursor finalizers.
@ -97,6 +102,48 @@ def _make_socket_reader(sock):
return _SocketReader(sock)
# Statement types from the DESCRIBE response's first field. The server
# tells us exactly what it prepared; these are the two values that mean
# "this will produce rows".
_ST_SELECT = 2
_ST_ROUTINE = 56 # EXECUTE PROCEDURE / EXECUTE FUNCTION
# Transaction control run as SQL. JDBC reads the same three values off
# the describe and calls setTxBeginState / setTxEndState (IfxSqli, the
# TxStmt field). Both ``BEGIN`` and ``BEGIN WORK`` report 34; likewise
# the WORK-less spellings of the other two.
_ST_TX_BEGIN = 34
_ST_TX_COMMIT = 35
_ST_TX_ROLLBACK = 36
_TX_CONTROL_TYPES = frozenset({_ST_TX_BEGIN, _ST_TX_COMMIT, _ST_TX_ROLLBACK})
def _produces_result_set(statement_type: int, ncolumns: int) -> bool:
"""Whether a prepared statement needs a cursor opened for it.
This is JDBC's ``IfxSqli.isResultSet`` predicate, and it replaces a
first-word check for ``SELECT`` that got five ordinary forms wrong.
A leading comment of any of the three Informix flavours, a
parenthesized select, a parenthesized UNION, and a CTE were all
classified as DML, so the driver sent SQ_EXECUTE where the server
expected a cursor. The report back was ``-260 Cursor name already in
use``, which describes neither the cause nor anything the caller did.
The original comment justified the heuristic on the grounds that
``nfields`` can't distinguish these, because ``INSERT INTO t VALUES
(?)`` also describes a column. True, and irrelevant: ``statement_type``
distinguishes them exactly. That INSERT reports 6 with one field; every
SELECT form above reports 2. The value was being parsed into the
describe metadata and thrown away.
``EXECUTE PROCEDURE``/``FUNCTION`` (56) is the one type that depends on
the column count, since a routine may or may not return rows.
"""
if statement_type == _ST_SELECT:
return True
return statement_type == _ST_ROUTINE and ncolumns > 0
def _finalize_cursor(
conn_ref: weakref.ReferenceType,
state: list,
@ -120,13 +167,30 @@ def _finalize_cursor(
a list (not the cursor object itself) keeps the finalizer's closure
weak the cursor remains GC'd-able.
"""
from ._protocol import ProtocolError
if not state[0]:
return # nothing to release
conn = conn_ref()
if conn is None or conn.closed:
return
if conn._wire_lock.held_by_current_thread:
# GC fired on the very thread that is mid-statement. The
# non-blocking acquire below would *succeed* here — an RLock
# grants a reentrant acquire to its own owner — and we would
# send CLOSE/RELEASE into the middle of that statement, killing
# it with -208. Defer instead, exactly as for another thread.
#
# Refcounting hides this: a dropped cursor is freed at the drop,
# before the next statement. A cursor caught in a reference cycle
# waits for a collection instead, and cycles are ordinary — any
# traceback holding a cursor makes one.
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
_log.debug(
"cursor finalizer: GC ran on the thread holding the wire lock; "
"enqueued CLOSE+RELEASE for deferred cleanup on conn %s",
id(conn),
)
return
if not conn._wire_lock.acquire(blocking=False):
# Another thread is mid-operation on this connection. Don't
# deadlock; instead, hand the cleanup bytes to the connection's
@ -147,19 +211,33 @@ def _finalize_cursor(
conn._drain_to_eot()
conn._send_pdu(_RELEASE_PDU)
conn._drain_to_eot()
except (ProtocolError, OSError) as exc:
# Wire desync during cleanup — same doctrine as
# ``_raise_sq_err``: the wire is unrecoverable, force-close
# the connection. Asymmetric handling of the same failure
# mode would be a Hamilton smell.
_log.warning(
"cursor finalizer: wire desync during cleanup; "
"force-closing connection: %r",
exc,
)
conn._closed = True
with contextlib.suppress(Exception):
conn._sock.close()
except WIRE_ERRORS as exc:
if getattr(exc, "sqlcode", None) is not None:
# The *server* rejected the cleanup — typically a stale
# CLOSE for a cursor it no longer has, which answers
# -267 "the transaction has been rolled back". That is
# an OperationalError, which is in WIRE_ERRORS, so this
# branch used to force-close a perfectly healthy
# connection over a no-op. ``_raise_sq_err`` self-drains
# the trailing SQ_EOT, so the wire is still aligned.
# Same distinction as _drain_pending_cleanup.
_log.debug(
"cursor finalizer: server rejected cleanup (stale): %r",
exc,
)
else:
# No sqlcode: the socket died or framing desynced and we
# can no longer say where a response ends. Force-close —
# same doctrine as ``_raise_sq_err``. Asymmetric handling
# of the same failure mode would be a Hamilton smell.
_log.warning(
"cursor finalizer: wire desync during cleanup; "
"force-closing connection: %r",
exc,
)
conn._closed = True
with contextlib.suppress(Exception):
conn._sock.close()
except InterfaceError:
# Connection was closed by another thread between our
# ``conn.closed`` check above and the actual write. No-op:
@ -169,17 +247,103 @@ def _finalize_cursor(
conn._wire_lock.release()
_ASCII_DIGITS = frozenset("0123456789")
def _rewrite_numeric_to_qmark(sql: str) -> str:
"""Convert ``:1`` / ``:2`` placeholders (paramstyle="numeric") to ``?``.
Informix's wire protocol uses ``?`` natively. Since we expose
``paramstyle="numeric"`` in the public API (matches Informix
ESQL/C convention), we rewrite before sending. Trivial cases only
strings and comments are NOT escaped, so SQL containing literal
``:1`` inside string literals will be wrongly substituted. Phase 5
can add a proper SQL tokenizer.
Informix's wire protocol uses ``?`` natively, and we advertise
``paramstyle="numeric"`` to match the ESQL/C convention, so the
placeholders are rewritten on the way out.
This used to be ``re.sub(r":(\\d+)", "?", sql)``, which cannot see a
string literal and so rewrote the contents of one. Any ``HH:MM``
time, any URL with a port, any aspect ratio::
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
stored as 'http://host?/x'
Nothing about that is opt-in. The rewrite runs whenever a statement
has parameters, whatever placeholder style the caller actually used,
so writing ``?`` everywhere and never touching numeric style did not
protect you. It also changed the placeholder *count* while
``num_qmarks`` was still computed from ``len(params)``, leaving the
driver and the server disagreeing about how many binds exist.
So: a single pass that substitutes only outside quotes and comments.
The lexical rules are Informix's own, measured against 12.10, 14.10
and 15 rather than assumed from standard SQL:
* ``''`` doubling escapes a quote inside ``'...'``. A backslash does
**not** escape anything; ``'a\'b'`` is an unterminated string and
the server answers ``-282``. A Postgres-style scanner that honours
``\'`` would desync here and corrupt everything after it.
* ``"..."`` is a delimited identifier or a string depending on
``DELIMIDENT``. Either way its contents are not ours to touch.
* ``--`` runs to end of line, ``/* */`` does **not** nest (the first
``*/`` closes it; nesting is a syntax error), and ``{ }`` is a
comment too.
* ``::`` is the cast operator and is stepped over as a unit, so it
can never be read as the start of a placeholder.
An unterminated quote or comment consumes the rest of the string and
substitutes nothing further. That is deliberate: under-substituting
leaves the server to reject SQL that was already malformed, while
guessing would corrupt a literal.
"""
return _NUMERIC_PLACEHOLDER_RE.sub("?", sql)
if ":" not in sql:
return sql
out: list[str] = []
i = 0
n = len(sql)
while i < n:
ch = sql[i]
if ch in ("'", '"'):
j = i + 1
while j < n:
if sql[j] == ch:
if j + 1 < n and sql[j + 1] == ch:
j += 2 # doubled quote, still inside
continue
j += 1
break
j += 1
out.append(sql[i:j])
i = j
elif ch == "-" and sql.startswith("--", i):
j = sql.find("\n", i)
j = n if j == -1 else j
out.append(sql[i:j])
i = j
elif ch == "/" and sql.startswith("/*", i):
j = sql.find("*/", i + 2)
j = n if j == -1 else j + 2
out.append(sql[i:j])
i = j
elif ch == "{":
j = sql.find("}", i)
j = n if j == -1 else j + 1
out.append(sql[i:j])
i = j
elif ch == ":":
if sql.startswith("::", i):
out.append("::")
i += 2
continue
j = i + 1
while j < n and sql[j] in _ASCII_DIGITS:
j += 1
if j > i + 1:
out.append("?")
i = j
else:
out.append(ch)
i += 1
else:
out.append(ch)
i += 1
return "".join(out)
def _generate_cursor_name() -> str:
@ -210,6 +374,9 @@ class Cursor:
# manipulation. Two-mode cursor; the same surface API works
# for both.
self._scrollable = scrollable
# Inherited from the connection, overridable per cursor. See
# informix_db.rows for what this costs and why it is opt-in.
self.row_factory = connection.row_factory
self._description: list[tuple] | None = None
self._columns: list[ColumnInfo] = []
self._column_readers: list[tuple] | None = None # Phase 37
@ -247,6 +414,11 @@ class Cursor:
# from. Empirically the server accepts 0 here even when a real
# ID was assigned, so this is best-effort tracking.
self._statement_id: int = 0
# DESCRIBE's statement-type field. Decides whether a cursor is
# opened -- see _produces_result_set.
self._statement_type: int = 0
# Per-result-set row class when row_factory is set, else None.
self._row_class: type | None = None
# Phase 10: smart-LOB read via ``lotofile(col, path, 'client')``.
# The server orchestrates a SQ_FILE (98) protocol where it tells
# us to "open file X, write these bytes, close". We emulate the
@ -331,6 +503,15 @@ class Cursor:
def _execute_under_wire_lock(self, sql: str, params: tuple) -> None:
"""Wire-bound body of ``execute``. Caller MUST hold ``_wire_lock``."""
# Someone else's scrollable cursor owns the session's statement
# slot -- refuse rather than draw a -285 and destroy their cursor.
self._conn._check_scroll_cursor_conflict(self)
# Our own previous scrollable cursor is still open server-side.
# Re-executing over the top of it collides exactly the same way,
# so close it first. This is the one caller allowed past the
# check above, and it is only safe because of this.
if self._scrollable and self._server_cursor_open:
self._close_server_cursor()
# Reset previous-execute state.
self._description = None
self._columns = []
@ -339,27 +520,37 @@ class Cursor:
self._rowcount = -1
self._rows = []
self._row_index = -1 # before-first-row
self._row_class = None
self._statement_type = 0
self._statement_already_done = False
# Step 1: PREPARE — send SQL with numQmarks = len(params).
# statement_boundary: nothing is open server-side yet, so this is
# the one safe moment to flush a finalizer's deferred cleanup.
self._conn._send_pdu(
self._build_prepare_pdu(sql, num_qmarks=len(params)),
statement_boundary=True,
)
self._read_describe_response()
# On a logged DB in non-autocommit mode, the server requires an
# explicit SQ_BEGIN before the first DML in each transaction.
# _ensure_transaction is a no-op for autocommit / unlogged DBs,
# and idempotent within an open transaction.
self._conn._ensure_transaction()
#
# This runs *after* the describe, matching JDBC's
# initiateTransaction placement, because until the describe lands
# we don't know whether the caller's statement is itself
# transaction control. Opening a transaction on their behalf and
# then executing their BEGIN WORK gets -535, "already in
# transaction" — the driver competing with the user for the same
# job and the user losing.
if self._statement_type not in _TX_CONTROL_TYPES:
self._conn._ensure_transaction()
# Step 1: PREPARE — send SQL with numQmarks = len(params).
self._conn._send_pdu(self._build_prepare_pdu(sql, num_qmarks=len(params)))
self._read_describe_response()
# Branch on the SQL keyword. We can't use ``self._columns`` /
# ``nfields`` here because a parameterized INSERT also returns
# a non-empty DESCRIBE (server describes the would-be inserted
# row's columns). The SQL-keyword heuristic is what JDBC effectively
# does too via its IfxStatement / IfxPreparedStatement subclassing.
first_word = sql.lstrip().split(None, 1)[0].upper() if sql.strip() else ""
is_select = first_word == "SELECT"
if is_select:
# Ask the server what it just prepared, rather than guessing from
# the first word of the SQL.
if _produces_result_set(self._statement_type, len(self._columns)):
if params:
self._execute_select_with_params(params)
else:
@ -369,6 +560,13 @@ class Cursor:
else:
self._execute_dml()
self._row_class = self._resolve_row_class()
# The statement succeeded. If it was transaction control, the
# server's transaction state just changed and the connection has
# to know, or commit() and rollback() silently do nothing.
self._note_transaction_control()
# SELECT path: position cursor before the first row so the next
# ``fetchone()`` returns ``rows[0]``. DML paths leave _row_index
# at -1 too (no rows to iterate).
@ -377,6 +575,98 @@ class Cursor:
if self._description is not None:
self._row_index = -1
def _resolve_row_class(self) -> type | None:
"""Pick the class this result set's rows are handed back as.
``row_factory`` is either :class:`informix_db.Row` (or a subclass),
in which case the per-shape class is built and cached from the
column names, or any callable taking the name tuple and returning
something that takes a values tuple.
Returns ``None`` when no factory is set, which is the default and
keeps plain tuples on the hot path at zero cost.
"""
factory = self.row_factory
if factory is None or self._description is None:
return None
names = tuple(d[0] for d in self._description)
if isinstance(factory, type) and issubclass(factory, Row):
return make_row_class(names)
return factory(names)
def _note_transaction_control(self) -> None:
"""Sync the connection's transaction flag after a successful execute.
``BEGIN WORK`` run through ``execute()`` opens a real transaction
on the server, and the connection had no idea. With autocommit on
nothing stopped it reaching the server, so ``_in_transaction``
stayed False while a transaction was open and both ``commit()``
and ``rollback()`` are guarded by that flag. ``rollback()``
returned successfully having sent nothing, and the rows it was
asked to discard were still there.
The pool reads the same flag to decide whether a returned
connection needs cleaning up, so the connection went back into
circulation holding an open transaction and its locks.
Called only on the success path: a statement that failed did not
change the server's transaction state.
"""
statement_type = self._statement_type
if statement_type == _ST_TX_BEGIN:
self._conn._in_transaction = True
elif statement_type in (_ST_TX_COMMIT, _ST_TX_ROLLBACK):
self._conn._in_transaction = False
def _close_server_cursor(self) -> None:
"""Free the server-side scrollable cursor. Caller MUST hold ``_wire_lock``.
Best-effort: a wire failure here is swallowed. Both callers are
already past the point where reporting it would help one is
closing the cursor, the other is about to run a new statement
that will report its own failure if the wire is really gone.
"""
try:
self._conn._send_pdu(self._build_close_pdu())
self._drain_to_eot()
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
except Exception:
pass
self._server_cursor_open = False
self._conn._open_scroll_cursor = None
def _release_after_failure(self, *, close_cursor: bool = False) -> None:
"""Best-effort server-side cleanup after a statement fails.
A statement that failed is still allocated. Skipping the release
bricks the connection: the next PREPARE collides with the leaked
one and every subsequent call returns a nonsense error whose
offset points back at the *failed* SQL. A duplicate-key violation
is about the most ordinary error an application can hit, so "one
constraint violation kills the connection" was easy to reach and
hard to attribute.
CLOSE and RELEASE get **separate** suppressions. Putting both in
one ``with contextlib.suppress(Exception)`` block reads as "clean
up both", but a CLOSE that raises skips the RELEASE entirely —
and RELEASE is the one that matters. Failing to close a cursor
wastes a handle; failing to release the statement is what breaks
the next call.
Everything here is swallowed rather than propagated. If the wire
is genuinely desynced then the cleanup fails too, and the caller
is far better served by the real SQL error than by a secondary
failure from the cleanup path.
"""
if close_cursor:
with contextlib.suppress(Exception):
self._conn._send_pdu(self._build_close_pdu())
self._drain_to_eot()
with contextlib.suppress(Exception):
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
def _execute_select_with_params(self, params: tuple) -> None:
"""Parameterized SELECT: SQ_BIND → CURNAME+NFETCH → drain → CLOSE+RELEASE.
@ -395,12 +685,20 @@ class Cursor:
try:
pdu = self._build_bind_only_pdu(params)
except Exception:
with contextlib.suppress(Exception):
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
self._release_after_failure()
raise
self._conn._send_pdu(pdu)
self._drain_to_eot()
try:
self._drain_to_eot()
except Exception:
# The server can reject the BIND itself — a value whose type
# doesn't match the described parameter, a bind against a
# statement the server has since invalidated. This was the
# last unguarded door of the six: the build was covered and
# the drain was not, so a server-side bind rejection left the
# statement allocated and the *next* execute() failed instead.
self._release_after_failure()
raise
# Now open the cursor and fetch — the bound values are in scope
# for the prepared statement.
self._execute_select()
@ -423,9 +721,23 @@ class Cursor:
self._conn._send_pdu(
self._build_curname_scroll_open_pdu(cursor_name)
)
self._drain_to_eot()
try:
self._drain_to_eot()
except Exception:
# Opening a scrollable cursor fails like any other
# statement — a bad ORDER BY, a permission error, a table
# dropped between PREPARE and OPEN. This branch had no
# guard at all, and it is the worst place to lack one:
# the GC-time finalizer is armed on the line *after* the
# drain, so a failure here left the statement allocated
# with no fallback whatsoever to reclaim it.
self._release_after_failure(close_cursor=True)
raise
self._server_cursor_open = True
self._finalizer_state[0] = True # arm the GC-time fallback
# The connection needs to know its statement slot is taken,
# so the next statement can refuse instead of drawing -285.
self._conn._open_scroll_cursor = weakref.ref(self)
self._scroll_total_rows = None
return # don't close; cursor stays live for SQ_SFETCH
# Phase 35: NFETCH loop — keep fetching until a response yields
@ -435,20 +747,32 @@ class Cursor:
# This bug was latent for ~30 phases because no test used a
# large enough result set to trigger it.
self._conn._send_pdu(self._build_curname_nfetch_pdu(cursor_name))
rows_before = len(self._rows)
self._read_fetch_response()
rows_received = len(self._rows) - rows_before
while rows_received > 0:
self._conn._send_pdu(self._build_nfetch_pdu())
try:
rows_before = len(self._rows)
self._read_fetch_response()
rows_received = len(self._rows) - rows_before
# Dereference BYTE/TEXT blob descriptors BEFORE CLOSE — the
# locators are only valid while the cursor is open. No-op when
# no BYTE/TEXT columns are present.
self._dereference_blob_columns()
while rows_received > 0:
self._conn._send_pdu(self._build_nfetch_pdu())
rows_before = len(self._rows)
self._read_fetch_response()
rows_received = len(self._rows) - rows_before
# Dereference BYTE/TEXT blob descriptors BEFORE CLOSE — the
# locators are only valid while the cursor is open. No-op when
# no BYTE/TEXT columns are present.
self._dereference_blob_columns()
except Exception:
# A raise anywhere in the fetch loop leaves the cursor and the
# statement allocated server-side. The GC-time finalizer only
# covers scrollable cursors (it is armed above, in the branch
# that returns early), so this path needs its own cleanup —
# otherwise a mid-fetch failure leaks and the next statement
# collides with it. Same failure mode as the DML path; see
# _release_after_failure for what that looks like from the
# caller's side.
self._release_after_failure(close_cursor=True)
raise
self._conn._send_pdu(self._build_close_pdu())
self._drain_to_eot()
@ -834,12 +1158,17 @@ class Cursor:
try:
pdu = self._build_bind_execute_pdu(params)
except Exception:
with contextlib.suppress(Exception):
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
self._release_after_failure()
raise
self._conn._send_pdu(pdu)
self._drain_to_eot()
try:
self._drain_to_eot()
except Exception:
# The statement is still allocated server-side even though it
# failed. See _execute_dml for why skipping this bricks the
# connection.
self._release_after_failure()
raise
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
@ -859,7 +1188,27 @@ class Cursor:
let the optimization-looking response confuse you.
"""
self._conn._send_pdu(self._build_execute_pdu())
self._drain_to_eot() # reads DONE + COST + EOT, populates rowcount
try:
self._drain_to_eot() # reads DONE + COST + EOT, populates rowcount
except Exception:
# A statement that FAILS is still allocated on the server, and
# skipping the release here bricked the whole connection: the
# next PREPARE collided with the leaked statement and every
# subsequent call returned a nonsense error (-255 "Not in
# transaction" in autocommit, -285 otherwise) whose offset
# pointed back at the *failed* SQL, not the new statement.
#
# A duplicate-key violation is about the most ordinary error an
# application can hit, so "one constraint violation kills the
# connection" was easy to reach and hard to attribute. Send the
# release, then re-raise the original error.
#
# Suppressed rather than propagated: if the wire is genuinely
# desynced the release will fail too, and the caller is far
# better served by the real SQL error than by a secondary
# failure from the cleanup path.
self._release_after_failure()
raise
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
@ -916,7 +1265,10 @@ class Cursor:
f"expected {first_len} (matching set [0])"
)
# Detect SELECT — not supported in executemany.
# Cheap pre-flight reject for the obvious case, so the common
# mistake costs no round-trip. The authoritative check is after
# PREPARE, below — this one shares the first-word heuristic's
# blind spots (leading comments, CTEs, parenthesized selects).
first_word = operation.lstrip().split(None, 1)[0].upper() if operation.strip() else ""
if first_word == "SELECT":
raise NotSupportedError("executemany on SELECT is not supported")
@ -927,6 +1279,9 @@ class Cursor:
# under the wire lock — N rows commit atomically with respect
# to other threads on the connection.
with self._conn._wire_lock:
self._conn._check_scroll_cursor_conflict(self)
if self._scrollable and self._server_cursor_open:
self._close_server_cursor()
# Reset per-execute state.
self._description = None
self._columns = []
@ -934,27 +1289,61 @@ class Cursor:
self._rowcount = -1
self._rows = []
self._row_index = -1
self._statement_type = 0
self._statement_already_done = False
# Logged-DB transaction guard — same as execute(). Idempotent
# within an open transaction.
self._conn._ensure_transaction()
# PREPARE once.
self._conn._send_pdu(
self._build_prepare_pdu(sql, num_qmarks=first_len)
self._build_prepare_pdu(sql, num_qmarks=first_len),
statement_boundary=True,
)
self._read_describe_response()
# Now the server has told us what it prepared. A result-set
# statement that slipped past the first-word check above --
# a CTE, a leading comment, a parenthesized select -- would
# otherwise be executed N times down the DML path, which
# opens no cursor and answers -260.
if _produces_result_set(self._statement_type, len(self._columns)):
self._release_after_failure()
raise NotSupportedError(
"executemany on a statement that returns rows is not "
"supported"
)
# Logged-DB transaction guard — same as execute(), and for the
# same reason placed after the describe rather than before it.
if self._statement_type not in _TX_CONTROL_TYPES:
self._conn._ensure_transaction()
# Phase 33: pipeline — build all BIND+EXECUTE PDUs first
# (Python work, no I/O), then send them back-to-back, then
# drain all responses. Eliminates the per-row round-trip
# the older serial loop paid.
pdus = [
self._build_bind_execute_pdu(tuple(p)) for p in seq
]
for pdu in pdus:
self._conn._send_pdu(pdu)
try:
pdus = [
self._build_bind_execute_pdu(tuple(p)) for p in seq
]
for pdu in pdus:
self._conn._send_pdu(pdu)
except Exception:
# Encoding a row can fail client-side (DataError for a
# value the connection's codec can't represent), and it
# fails here — after the PREPARE, before anything is
# drained. Escaping without the RELEASE leaks the
# statement server-side, and the next PREPARE then
# collides with it: every later call on this connection
# returns a nonsense error pointing at the *previous*
# SQL. ``_execute_dml_with_params`` guards the identical
# case for the single-row path; the pipelined path was
# simply missed.
#
# A send failure lands here too. The wire is likely
# already unusable in that case, but attempting the
# release costs nothing and the original error is what
# propagates either way.
self._release_after_failure()
raise
# Drain N responses. The first error is captured but we
# still drain the rest (they're SQ_ERRs for the aborted
@ -1026,7 +1415,8 @@ class Cursor:
self._row_index = len(self._rows) # past-last
return None
self._row_index = nxt
return self._rows[nxt]
row = self._rows[nxt]
return self._row_class(row) if self._row_class is not None else row
def fetchmany(self, size: int | None = None) -> list[tuple]:
self._check_open()
@ -1057,6 +1447,8 @@ class Cursor:
return []
start = self._row_index + 1
out = self._rows[start:]
if self._row_class is not None:
out = [self._row_class(r) for r in out]
self._row_index = len(self._rows)
return list(out)
@ -1237,7 +1629,7 @@ class Cursor:
if scrolltype == 4 or is_last_probe:
# SFETCH(LAST) — TUPID == total row count
self._scroll_total_rows = self._last_tupid
return row
return self._row_class(row) if self._row_class is not None else row
def close(self) -> None:
"""Close the cursor.
@ -1249,20 +1641,8 @@ class Cursor:
if self._closed:
return
if self._scrollable and self._server_cursor_open:
# Phase 27: hold the wire lock during CLOSE+RELEASE so we
# don't interleave with another thread's pending op on the
# connection. Best-effort: any wire failure here is
# swallowed (the caller is closing; we don't want to mask
# whatever caused them to close).
try:
with self._conn._wire_lock:
self._conn._send_pdu(self._build_close_pdu())
self._drain_to_eot()
self._conn._send_pdu(self._build_release_pdu())
self._drain_to_eot()
except Exception:
pass
self._server_cursor_open = False
with self._conn._wire_lock:
self._close_server_cursor()
# Phase 28: explicit close has handled the server-side resources
# (or tried to). Disarm the finalizer so it doesn't fire later
# for nothing — and clear the state flag as a belt-and-suspenders
@ -1566,6 +1946,7 @@ class Cursor:
elif tag == MessageType.SQ_DESCRIBE:
self._columns, meta = parse_describe(reader)
self._statement_id = meta.get("statement_id", 0)
self._statement_type = meta.get("statement_type", 0)
self._description = (
[c.to_description_tuple() for c in self._columns] if self._columns else None
)
@ -1697,7 +2078,6 @@ class Cursor:
(e.g. table or column name for "not found" errors). Empty for
most syntax errors.
"""
from ._protocol import ProtocolError
sqlcode = reader.read_short()
isamcode = reader.read_short()
@ -1715,7 +2095,7 @@ class Cursor:
if name_len & 1:
reader.read_exact(1) # pad to even
near_token = raw.rstrip(b"\x00").decode("iso-8859-1", errors="replace")
except (ProtocolError, OSError):
except WIRE_ERRORS:
pass
# Drain remaining bytes until SQ_EOT. Phase 28: a (ProtocolError,
# OSError) during drain means the wire is in an unknown state —
@ -1727,7 +2107,7 @@ class Cursor:
t = reader.read_short()
if t == MessageType.SQ_EOT:
break
except (ProtocolError, OSError):
except WIRE_ERRORS:
with contextlib.suppress(Exception):
self._conn.close()

180
src/informix_db/rows.py Normal file
View File

@ -0,0 +1,180 @@
"""Rows that can be read by position, by column name, or by attribute.
PEP 249 only requires a sequence, and a sequence is what the driver
returns by default. That is fine for ``SELECT a, b`` and steadily worse
as the projection grows: ``row[11]`` tells a reader nothing, and stays
correct only until somebody adds a column in the middle.
Opting in with ``row_factory=Row`` gives the shape ``pyodbc`` and
``mssql-python`` provide, all three at once::
conn = informix_db.connect(..., row_factory=informix_db.Row)
cur.execute("SELECT tabid, tabname FROM systables")
row = cur.fetchone()
row[0], row["tabname"], row.tabname
It is opt-in rather than the default because it is not free, and the
driver's whole argument is that pure Python can stay within noise of the
C driver on bulk fetch. Measured on a 20,000-row five-column fetch:
**37.2 ms with tuples, 40.8 ms with Row**, so about 9%. Defaulting it on
would move the published 1.05-1.15x ratio against IfxPy to roughly
1.15-1.25x, which is not a trade to make on everybody's behalf.
Where the 9% goes, against a plain tuple: about 30 ns per row to
construct, about 39 ns per ``row[0]`` because supporting ``row["name"]``
means ``__getitem__`` is a Python method rather than C-level tuple
indexing, and about 13 ns per unpack since CPython's fast path for
``a, b = row`` applies to exact tuples and not to subclasses.
Worth it for readability in application code. Not worth paying in a bulk
export that never looks at a column by name, which is exactly why it is
a choice.
``Row`` subclasses ``tuple``, so ``row == (1, "x")`` is still true and
existing code keeps working unchanged.
**A column always beats a method of the same name.** ``tuple`` defines
``count`` and ``index``; this class adds ``keys``, ``_asdict`` and
``_fields``. A column named any of those would otherwise resolve to the
method and hand back a bound method instead of a value, which is exactly
the shape of the framing bugs this driver spent a fortnight removing: a
plausible-looking wrong answer, in silence. So every such name gets a
descriptor and the column wins, and the reserved set is *computed* from
the class rather than hand-listed, because hand-listing it is what
missed ``keys``, ``_asdict`` and ``_fields`` the first time round. The
methods stay reachable through the class: ``tuple.count(row, x)``,
``Row.keys(row)``.
The machinery itself is name-mangled (``__fields`` / ``__map``) so that
shadowing ``_fields`` cannot break ``repr`` or ``_asdict``.
Column names come from ``cursor.description``. Informix folds unquoted
identifiers to lower case, so ``SELECT Config_Key`` is reachable as
``row.config_key``. Expressions get server-generated names that are not
Python identifiers, like ``(count(*))``; those are reachable by
subscript but not as attributes. Duplicate names resolve to the first
occurrence, matching ``pyodbc``. Dunder names cannot be shadowed and
stay subscript-only, which no real schema should notice.
"""
from __future__ import annotations
import operator
from functools import lru_cache
from typing import ClassVar
__all__ = ["Row", "make_row_class"]
class Row(tuple):
"""A result row addressable by position, name, or attribute.
Used as a ``row_factory``. The concrete class handed to each result
set is a subclass carrying that query's column names, built by
:func:`make_row_class`.
"""
__slots__ = ()
# Name-mangled to ``_Row__fields`` / ``_Row__map`` so that a column
# called "_fields" can be shadowed without breaking the machinery
# that reads it. Set on the per-result-set subclass.
__fields: ClassVar[tuple[str, ...]] = ()
__map: ClassVar[dict[str, int]] = {}
def __getitem__(self, key):
# ``key.__class__ is str`` rather than isinstance: this runs on
# every subscript.
if key.__class__ is str:
try:
return tuple.__getitem__(self, self.__map[key])
except KeyError:
raise KeyError(
f"no column named {key!r}; this row has "
f"{list(self.__fields)}"
) from None
return tuple.__getitem__(self, key)
def __getattr__(self, name):
# Only reached when normal attribute lookup has already failed,
# so this costs nothing for names that do not collide.
try:
return tuple.__getitem__(self, self.__map[name])
except KeyError:
raise AttributeError(
f"no column named {name!r}; this row has "
f"{list(self.__fields)}"
) from None
@property
def _fields(self) -> tuple[str, ...]:
"""Column names, in select order. Mirrors ``namedtuple._fields``."""
return self.__fields
def keys(self) -> tuple[str, ...]:
"""Column names, in select order."""
return self.__fields
def _asdict(self) -> dict:
"""A plain ``dict`` of the row.
On duplicate column names the last occurrence wins here, while
subscript access gives the first. A dict cannot represent both,
and quietly dropping a duplicate is better than raising on a
query that is otherwise fine.
"""
return dict(zip(self.__fields, self, strict=True))
def __repr__(self) -> str:
fields = self.__fields
if len(fields) != len(self):
return tuple.__repr__(self)
body = ", ".join(
f"{name}={value!r}"
for name, value in zip(fields, self, strict=True)
)
return f"Row({body})"
def __reduce__(self):
# The per-result-set class is created at runtime and cannot be
# pickled by reference, so rebuild it from the field names.
return (_rebuild_row, (self.__fields, tuple(self)))
# Every non-dunder attribute a Row already answers to. A column with one
# of these names gets a descriptor so the column wins. Computed, not
# hand-listed: the hand-listed version covered ``count`` and ``index``
# and silently missed ``keys``, ``_asdict`` and ``_fields``.
_RESERVED = frozenset(
name for name in dir(Row) if not name.startswith("__")
) - {"_Row__fields", "_Row__map"}
def _rebuild_row(fields: tuple[str, ...], values: tuple):
return make_row_class(fields)(values)
@lru_cache(maxsize=256)
def make_row_class(fields: tuple[str, ...]) -> type[Row]:
"""Build (and cache) the row class for one column-name shape.
Cached because a class per ``execute()`` would put a ``type()`` call
on the path of every small query, and applications run the same
handful of statement shapes over and over. Keyed on the names alone,
so two queries selecting the same columns share a class.
"""
namespace: dict = {
"__slots__": (),
"_Row__fields": fields,
# First occurrence wins on duplicates, matching pyodbc. Building
# the map in reverse and letting earlier entries overwrite later
# ones is the shortest way to say that.
"_Row__map": {
name: i for i, name in reversed(list(enumerate(fields)))
},
}
for i, name in enumerate(fields):
if name in _RESERVED:
# A column of this name would otherwise resolve to a method.
namespace[name] = property(operator.itemgetter(i))
return type("Row", (Row,), namespace)

265
tests/test_async_threads.py Normal file
View File

@ -0,0 +1,265 @@
"""The async layer used a thread pool it shared with the whole process.
``asyncio.to_thread`` runs on the event loop's default executor. That
executor belongs to the process, not to us, and it is sized from the CPU
count ``min(32, cpu_count + 4)``, which is six threads on a two-CPU
container. Every blocking call in ``informix_db.aio`` went through it.
Two consequences, both silent.
**Cancelled calls hold their threads.** ``asyncio.to_thread`` cannot
interrupt a worker, so a cancelled await leaves the thread running the
wire call until the read timeout expires. Cancellation is ordinary in a
web app a client disconnect cancels the request task so a handful of
them pins every thread in the shared pool. Unrelated ``to_thread`` work
anywhere else in the process then stops dead, and so does the driver.
**Pool concurrency was capped by an unrelated number.** A pool with
``max_size=20`` on a two-CPU box ran six queries at a time, and nothing
said so.
Each connection now owns one thread. That is the right size rather than
a compromise: the sync connection serializes every wire operation on its
own lock, so a second thread could do nothing but wait for the first. It
also avoids a deadlock that a shared pool-sized executor invites with
N threads and N connections, N tasks blocked in ``acquire`` occupy every
thread while the connection they wait for is held by a task that now
needs a thread to finish and release it.
"""
from __future__ import annotations
import asyncio
import threading
import time
from concurrent.futures import ThreadPoolExecutor
import pytest
from informix_db import aio
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _kw(conn_params: ConnParams) -> dict:
return {
"host": conn_params.host,
"port": conn_params.port,
"user": conn_params.user,
"password": conn_params.password,
"database": conn_params.database,
"server": conn_params.server,
"connect_timeout": 10.0,
"read_timeout": 20.0,
"autocommit": True,
}
@pytest.mark.asyncio
async def test_queries_run_with_the_default_executor_fully_held(
conn_params: ConnParams,
) -> None:
"""The starvation case, with real queries. Four unrelated jobs occupy
every thread of a four-worker default executor; the driver must not
be waiting behind them."""
loop = asyncio.get_running_loop()
previous = loop._default_executor
loop.set_default_executor(ThreadPoolExecutor(max_workers=4))
conns = [await aio.connect(**_kw(conn_params)) for _ in range(4)]
cursors = [await c.cursor() for c in conns]
gate = threading.Event()
hogs = [
asyncio.create_task(asyncio.to_thread(gate.wait, 30)) for _ in range(4)
]
await asyncio.sleep(0.3)
async def query(cursor, i: int):
await cursor.execute(f"SELECT FIRST 1 tabid + {i} FROM systables")
return await cursor.fetchone()
try:
rows = await asyncio.wait_for(
asyncio.gather(*(query(c, i) for i, c in enumerate(cursors))),
timeout=10.0,
)
assert len(rows) == 4
assert all(r is not None for r in rows)
finally:
gate.set()
await asyncio.gather(*hogs, return_exceptions=True)
for c in conns:
await c.close()
# set_default_executor rejects None, which is what the loop
# starts with before anything has used to_thread.
loop.set_default_executor(previous or ThreadPoolExecutor())
@pytest.mark.asyncio
async def test_each_connection_gets_its_own_thread(
conn_params: ConnParams,
) -> None:
a = await aio.connect(**_kw(conn_params))
b = await aio.connect(**_kw(conn_params))
try:
assert a._executor is not b._executor
assert a._executor._max_workers == 1, (
"more than one thread per connection cannot help — the wire "
"lock serializes them anyway"
)
finally:
await a.close()
await b.close()
@pytest.mark.asyncio
async def test_cursor_runs_on_its_connections_thread(
conn_params: ConnParams,
) -> None:
"""A cursor must not fall back to the default executor, or half the
work goes back to being shared."""
conn = await aio.connect(**_kw(conn_params))
try:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
names: list[str] = []
await conn._run(lambda: names.append(threading.current_thread().name))
assert names[0].startswith("informix-conn")
assert cur._run.__self__ is conn
finally:
await conn.close()
@pytest.mark.asyncio
async def test_pool_reuses_one_thread_per_connection(
conn_params: ConnParams,
) -> None:
"""The executor lives on the sync connection, so a connection handed
out, returned, and handed out again keeps the same thread instead of
spawning one per acquire."""
pool = await aio.create_pool(**_kw(conn_params), min_size=1, max_size=2)
try:
seen = []
for _ in range(4):
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
assert await cur.fetchone() is not None
seen.append(id(conn._executor))
assert len(set(seen)) <= 2, (
f"expected at most one executor per pooled connection, saw "
f"{len(set(seen))}"
)
finally:
await pool.close()
@pytest.mark.asyncio
async def test_pool_concurrency_is_not_capped_by_the_default_executor(
conn_params: ConnParams,
) -> None:
"""Six concurrent pooled queries against a two-worker default
executor. Under the old arrangement at most two could run."""
loop = asyncio.get_running_loop()
previous = loop._default_executor
pool = await aio.create_pool(**_kw(conn_params), min_size=6, max_size=6)
loop.set_default_executor(ThreadPoolExecutor(max_workers=2))
try:
async def one(i: int):
async with pool.connection(timeout=15.0) as conn:
cur = await conn.cursor()
# Long enough that serialized execution would be obvious.
await cur.execute(
"SELECT FIRST 200 a.tabid FROM systables a, systables b"
)
return len(await cur.fetchall())
started = time.monotonic()
results = await asyncio.wait_for(
asyncio.gather(*(one(i) for i in range(6))), timeout=25.0
)
assert all(r > 0 for r in results)
assert len(results) == 6
elapsed = time.monotonic() - started
assert elapsed < 20.0, f"queries appear serialized ({elapsed:.1f}s)"
finally:
await pool.close()
# set_default_executor rejects None, which is what the loop
# starts with before anything has used to_thread.
loop.set_default_executor(previous or ThreadPoolExecutor())
@pytest.mark.asyncio
async def test_closing_a_connection_stops_its_thread(
conn_params: ConnParams,
) -> None:
"""One thread per connection is only affordable if the thread goes
away with the connection."""
before = threading.active_count()
conns = [await aio.connect(**_kw(conn_params)) for _ in range(5)]
for c in conns:
cur = await c.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
for c in conns:
await c.close()
del conns
# Threads exit asynchronously after shutdown; give them a moment.
for _ in range(50):
if threading.active_count() <= before + 1:
break
await asyncio.sleep(0.1)
assert threading.active_count() <= before + 1, (
f"connection threads outlived their connections "
f"({before} -> {threading.active_count()})"
)
@pytest.mark.asyncio
async def test_row_factory_reaches_the_async_paths(
conn_params: ConnParams,
) -> None:
"""The async layer wraps the sync cursor, so row_factory should flow
through untouched. Each of the four async fetch routes is a separate
call site, and async iteration goes through __anext__ rather than
__next__."""
import informix_db
conn = await aio.connect(row_factory=informix_db.Row, **_kw(conn_params))
try:
cur = await conn.cursor()
sql = "SELECT FIRST 2 tabid, tabname FROM systables ORDER BY tabid"
await cur.execute(sql)
one = await cur.fetchone()
assert one[0] == one["tabid"] == one.tabid
await cur.execute(sql)
assert [r.tabid for r in await cur.fetchall()] == [1, 2]
await cur.execute(sql)
assert all(r.tabname for r in await cur.fetchmany(2))
await cur.execute(sql)
assert [r.tabid async for r in cur] == [1, 2]
finally:
await conn.close()
@pytest.mark.asyncio
async def test_async_pool_forwards_the_row_factory(
conn_params: ConnParams,
) -> None:
import informix_db
pool = await aio.create_pool(
row_factory=informix_db.Row, min_size=1, max_size=2, **_kw(conn_params)
)
try:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute("SELECT FIRST 1 tabid FROM systables")
assert (await cur.fetchone()).tabid == 1
finally:
await pool.close()

View File

@ -0,0 +1,258 @@
"""Regression tests for pipelined executemany, scrollable cursors, and LOBs.
Found here by fuzzing: **a client-side encoding failure inside an
``executemany`` batch leaked the prepared statement.** The PDUs are built
after the PREPARE, so a value the connection's codec cannot represent
raises there before anything is drained and the exception escaped
without the RELEASE. The leaked statement then collided with the next
PREPARE and every later call on that connection failed with an error
pointing at the *previous* SQL.
``_execute_dml_with_params`` already guarded exactly this case for the
single-row path. The pipelined path was simply missed, which is the
recurring shape of these bugs: a hazard understood in one place and not
carried to its sibling.
The rest of this file is the coverage that proved the neighbouring paths
sound batch failures at every position, scroll boundaries, LOB sizes
kept so they stay that way. Each failure case ends in a health check,
because the interesting damage is never in the statement that failed.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=15.0,
read_timeout=30.0,
**kw,
)
def _assert_healthy(cur) -> None:
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, "connection unusable"
# ---------------------------------------------------------------------------
# executemany — the encoding-failure leak, and the neighbours
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [3, 50])
@pytest.mark.parametrize("position", ["first", "mid", "last"])
def test_encoding_failure_in_batch_does_not_leak(
conn_params: ConnParams, n: int, position: str
) -> None:
"""The bug. A value the codec can't encode raises after the PREPARE;
escaping without the RELEASE bricked the connection."""
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_enc (s VARCHAR(32))")
rows: list[tuple] = [(f"s{i}",) for i in range(n)]
rows[idx] = ("中文",) # not representable in iso-8859-1
with pytest.raises(Exception): # noqa: B017 — DataError or UnicodeError
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
# The connection must survive, repeatedly.
for _ in range(3):
_assert_healthy(cur)
with pytest.raises(Exception): # noqa: B017
cur.executemany("INSERT INTO t_em_enc VALUES (?)", rows)
_assert_healthy(cur)
@pytest.mark.parametrize("n", [2, 3, 10, 100])
@pytest.mark.parametrize("position", ["first", "mid", "last"])
def test_constraint_violation_in_batch_recovers(
conn_params: ConnParams, n: int, position: str
) -> None:
idx = {"first": 0, "mid": n // 2, "last": n - 1}[position]
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_dup (k INT PRIMARY KEY)")
cur.execute("INSERT INTO t_em_dup VALUES (?)", (10_000,))
rows = [(i,) for i in range(n)]
rows[idx] = (10_000,)
with pytest.raises(informix_db.Error):
cur.executemany("INSERT INTO t_em_dup VALUES (?)", rows)
_assert_healthy(cur)
# Whatever the partial-batch semantics, COUNT(*) and a full fetch
# must agree — a disagreement means the row decoder and the server
# have different ideas about what is in the table.
cur.execute("SELECT COUNT(*) FROM t_em_dup")
(counted,) = cur.fetchone()
cur.execute("SELECT k FROM t_em_dup")
assert counted == len(cur.fetchall())
@pytest.mark.parametrize("n", [1, 2, 3, 10, 100, 1000])
def test_executemany_inserts_exactly_n_rows(
conn_params: ConnParams, n: int
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_ok (k INT, v VARCHAR(16))")
cur.executemany(
"INSERT INTO t_em_ok VALUES (?, ?)",
[(i, f"v{i}") for i in range(n)],
)
cur.execute("SELECT COUNT(*) FROM t_em_ok")
assert cur.fetchone() == (n,)
def test_executemany_empty_and_single(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_em_edge (k INT)")
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [])
_assert_healthy(cur)
cur.executemany("INSERT INTO t_em_edge VALUES (?)", [(1,)])
cur.execute("SELECT COUNT(*) FROM t_em_edge")
assert cur.fetchone() == (1,)
# ---------------------------------------------------------------------------
# Scrollable cursors — boundaries get their own round-trip each
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [0, 1, 2, 5, 50, 300])
def test_scroll_boundaries(conn_params: ConnParams, n: int) -> None:
with _connect(conn_params, autocommit=True) as conn:
setup = conn.cursor()
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll")
setup.execute("CREATE TABLE t_scroll (k INT)")
try:
if n:
setup.executemany(
"INSERT INTO t_scroll VALUES (?)", [(i,) for i in range(n)]
)
cur = conn.cursor(scrollable=True)
try:
cur.execute("SELECT k FROM t_scroll ORDER BY k")
assert cur.fetch_first() == ((0,) if n else None)
assert cur.fetch_last() == ((n - 1,) if n else None)
if n >= 3:
assert cur.fetch_absolute(2) == (2,)
assert cur.fetch_prior() == (1,)
assert cur.fetch_relative(2) == (3,)
# Off both ends must be None — not a crash, not a wrap.
assert cur.fetch_absolute(n + 50) is None
cur.fetch_first()
assert cur.fetch_prior() is None
if n:
cur.fetch_first()
seen = [(0,)]
while (row := cur.fetchone()) is not None:
seen.append(row)
assert seen == [(i,) for i in range(n)]
finally:
cur.close()
_assert_healthy(setup)
finally:
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll")
def test_abandoned_scroll_cursors_do_not_leak(
conn_params: ConnParams,
) -> None:
"""Scroll cursors stay open server-side, so abandoning one leaks
unless the finalizer runs."""
with _connect(conn_params, autocommit=True) as conn:
setup = conn.cursor()
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll_ab")
setup.execute("CREATE TABLE t_scroll_ab (k INT)")
try:
setup.executemany(
"INSERT INTO t_scroll_ab VALUES (?)", [(i,) for i in range(20)]
)
for i in range(20):
c = conn.cursor(scrollable=True)
c.execute("SELECT k FROM t_scroll_ab ORDER BY k")
c.fetch_first()
if i % 2:
c.close()
else:
del c # rely on the finalizer
_assert_healthy(setup)
setup.execute("SELECT COUNT(*) FROM t_scroll_ab")
assert setup.fetchone() == (20,)
finally:
with contextlib.suppress(Exception):
setup.execute("DROP TABLE t_scroll_ab")
# ---------------------------------------------------------------------------
# Smart LOBs
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
"size", [0, 1, 255, 256, 1023, 1024, 4095, 4096, 65535, 65536]
)
def test_blob_round_trip_sizes(conn_params: ConnParams, size: int) -> None:
"""Sizes straddle the 4096-byte SQ_FILE chunk and the 64K mark."""
payload = bytes((i * 7 + size) % 256 for i in range(size))
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_blob_rt")
try:
cur.execute("CREATE TABLE t_blob_rt (k INT, b BLOB)")
except informix_db.Error:
pytest.skip("no sbspace configured; see make ifx-spaces")
try:
cur.write_blob_column(
"INSERT INTO t_blob_rt VALUES (?, BLOB_PLACEHOLDER)",
payload, (1,),
)
got = cur.read_blob_column(
"SELECT b FROM t_blob_rt WHERE k = ?", (1,)
)
if size == 0:
assert got in (b"", None)
else:
assert got == payload
_assert_healthy(cur)
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_blob_rt")
def test_failed_blob_read_recovers(conn_params: ConnParams) -> None:
"""The SQ_FILE path involves a server-side temp file; a failure part
way through must not strand the connection."""
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
for _ in range(5):
with contextlib.suppress(Exception):
cur.read_blob_column("SELECT b FROM t_no_such_blob_tbl", ())
_assert_healthy(cur)

View File

@ -0,0 +1,248 @@
"""Rewriting :N placeholders without rewriting the SQL around them.
We advertise ``paramstyle="numeric"`` to match Informix's ESQL/C
convention, and the wire protocol takes ``?``, so placeholders get
rewritten on the way out. That was ``re.sub(r":(\\d+)", "?", sql)``,
which cannot see a string literal and therefore rewrote the inside of
one::
UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?
stored as 'http://host?/x'
Any ``HH:MM`` time, any URL with a port, any aspect ratio, any
``key:value`` string. It wrote wrong data and said nothing.
Nothing about it was opt-in either. The rewrite runs whenever a
statement has parameters, whatever placeholder style the caller actually
used, so writing ``?`` everywhere and never touching numeric style did
not protect you. And it changed the placeholder *count* while
``num_qmarks`` was still computed from ``len(params)``, leaving driver
and server disagreeing about how many binds exist.
The lexical rules below are Informix's own, measured against 12.10,
14.10 and 15 rather than assumed from standard SQL. The one that matters
most is the backslash: ``'a\\'b'`` is an *unterminated string* to
Informix (``-282``), not an escaped quote. A scanner written to
Postgres habits would desync on it and corrupt everything after.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _rewrite_numeric_to_qmark as rewrite
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# Substitution happens where it should
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("sql", "expected"),
[
("SELECT * FROM t WHERE a = :1", "SELECT * FROM t WHERE a = ?"),
(
"SELECT * FROM t WHERE a = :1 AND b = :2",
"SELECT * FROM t WHERE a = ? AND b = ?",
),
("SELECT * FROM t WHERE a = :10", "SELECT * FROM t WHERE a = ?"),
# Already-? SQL is returned untouched, and cheaply.
("SELECT * FROM t WHERE a = ?", "SELECT * FROM t WHERE a = ?"),
("", ""),
],
)
def test_placeholders_are_rewritten(sql: str, expected: str) -> None:
assert rewrite(sql) == expected
# ---------------------------------------------------------------------------
# ...and nowhere else
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("label", "sql", "expected"),
[
(
"url-with-port",
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = :1",
"UPDATE jobs SET url = 'http://host:8080/x' WHERE id = ?",
),
(
"time-of-day",
"INSERT INTO log VALUES ('started at 09:15:30', :1)",
"INSERT INTO log VALUES ('started at 09:15:30', ?)",
),
(
"aspect-ratio",
"UPDATE t SET ratio = '16:9' WHERE id = :1",
"UPDATE t SET ratio = '16:9' WHERE id = ?",
),
(
"doubled-quote-escape",
"SELECT 'it''s 10:30' FROM t WHERE a = :1",
"SELECT 'it''s 10:30' FROM t WHERE a = ?",
),
(
"delimited-identifier",
'SELECT "col:1" FROM t WHERE a = :1',
'SELECT "col:1" FROM t WHERE a = ?',
),
(
"line-comment",
"-- ticket :99\nSELECT * FROM t WHERE a = :1",
"-- ticket :99\nSELECT * FROM t WHERE a = ?",
),
(
"block-comment",
"SELECT /* not :99 */ * FROM t WHERE a = :1",
"SELECT /* not :99 */ * FROM t WHERE a = ?",
),
(
"brace-comment",
"SELECT { not :99 } * FROM t WHERE a = :1",
"SELECT { not :99 } * FROM t WHERE a = ?",
),
(
"cast-operator",
"SELECT a::INT FROM t WHERE a = :1",
"SELECT a::INT FROM t WHERE a = ?",
),
(
"colon-not-a-placeholder",
"SELECT a FROM t WHERE b = ':x' AND c = :1",
"SELECT a FROM t WHERE b = ':x' AND c = ?",
),
(
"literal-after-placeholder",
"SELECT * FROM t WHERE a = :1 AND b = '10:30'",
"SELECT * FROM t WHERE a = ? AND b = '10:30'",
),
],
)
def test_quotes_and_comments_are_left_alone(
label: str, sql: str, expected: str
) -> None:
assert rewrite(sql) == expected, label
def test_backslash_does_not_escape_a_quote() -> None:
"""Informix answers -282 for ``'a\\'b'``: the backslash is an ordinary
character and the string is unterminated. A scanner that treated it
as an escape would think it was still inside the literal and stop
substituting, or worse, resume in the wrong place."""
sql = "SELECT 'a\\' FROM t WHERE x = :1"
# The quote closes at the character after the backslash, so :1 is
# outside the literal and gets substituted.
assert rewrite(sql) == "SELECT 'a\\' FROM t WHERE x = ?"
def test_unterminated_quote_substitutes_nothing_further() -> None:
"""Under-substituting leaves the server to reject SQL that was
already malformed. Guessing would corrupt a literal."""
assert rewrite("SELECT 'oops :1 FROM t") == "SELECT 'oops :1 FROM t"
def test_unterminated_block_comment_substitutes_nothing_further() -> None:
assert rewrite("SELECT /* oops :1 FROM t") == "SELECT /* oops :1 FROM t"
def test_block_comments_do_not_nest() -> None:
"""Measured: ``/* a /* b */ c */`` is a syntax error on all three
servers, so the first ``*/`` closes the comment. Treating them as
nesting would swallow live SQL."""
assert (
rewrite("SELECT /* a /* b */ :1 FROM t")
== "SELECT /* a /* b */ ? FROM t"
)
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
)
@pytest.mark.integration
def test_colon_literals_round_trip(conn_params: ConnParams) -> None:
"""The bug as a user meets it: the value stored is not the value
written. Both placeholder styles, because the rewrite ran for both."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw (id INT, s VARCHAR(60))")
cur.execute(
"INSERT INTO t_rw VALUES (?, 'http://host:8080/path')", (1,)
)
cur.execute("INSERT INTO t_rw VALUES (:1, 'at 09:15:30')", (2,))
cur.execute("SELECT id, s FROM t_rw ORDER BY id")
assert cur.fetchall() == [
(1, "http://host:8080/path"),
(2, "at 09:15:30"),
]
@pytest.mark.integration
def test_colon_literals_round_trip_through_executemany(
conn_params: ConnParams,
) -> None:
"""executemany rewrites unconditionally, so it had the bug too."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw2 (id INT, s VARCHAR(40))")
cur.executemany(
"INSERT INTO t_rw2 VALUES (?, '16:9')", [(1,), (2,)]
)
cur.execute("SELECT DISTINCT s FROM t_rw2")
assert cur.fetchall() == [("16:9",)]
@pytest.mark.integration
def test_comment_bearing_sql_still_binds(conn_params: ConnParams) -> None:
"""A leading comment reaches PREPARE now that classification asks the
server. Make sure the rewriter agrees and doesn't eat a placeholder
that follows one."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"/* report: daily */ SELECT FIRST 1 tabid FROM systables "
"WHERE tabid > :1",
(0,),
)
assert cur.fetchone() is not None
@pytest.mark.integration
def test_placeholder_count_matches_after_rewrite(
conn_params: ConnParams,
) -> None:
"""The old regex could add placeholders the driver never counted,
leaving num_qmarks (from len(params)) disagreeing with the SQL. A
literal containing three colon-digit sequences is the shape that
used to break it."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rw3 (id INT, s VARCHAR(40))")
with contextlib.suppress(Exception):
cur.execute("DELETE FROM t_rw3")
cur.execute(
"INSERT INTO t_rw3 VALUES (:1, 'a:1 b:2 c:3')", (7,)
)
cur.execute("SELECT id, s FROM t_rw3")
assert cur.fetchone() == (7, "a:1 b:2 c:3")

View File

@ -0,0 +1,459 @@
"""Regression tests for error recovery, cursor lifecycle, and concurrency.
The type-matrix suite covers what a value looks like on the wire. This
covers everything around it: what happens after a statement fails, what
happens to abandoned cursors, and what happens when more than one thread
or task is involved.
Two bugs here were found by fuzzing, and both are the kind that never
show up in a single-threaded happy-path test:
* **A failed statement was never released.** Successful DML sent
PREPARE EXECUTE RELEASE; failing DML sent PREPARE EXECUTE and
stopped. The leaked statement collided with the next PREPARE, and
every subsequent call on that connection returned a nonsense error
whose offset pointed back at the *failed* SQL. A duplicate-key
violation about the most ordinary error an application can hit
killed the connection outright.
* **Cancelling a pool acquire leaked the connection.** ``asyncio.to_thread``
cannot interrupt its worker, so a cancelled waiter left the worker to
finish and hand back a connection nobody owned. Under HTTP load, where
client disconnects cancel request tasks precisely while they wait for
a connection, the pool dies one slot at a time.
The tests that follow assert on *data* wherever they can, not merely the
absence of an exception: a worker reads back a value only it wrote, so a
crossed wire fails even when nothing raises.
"""
from __future__ import annotations
import asyncio
import contextlib
import threading
import pytest
import informix_db
from informix_db import aio
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
# Bounded: a leaked statement used to manifest as a hang.
read_timeout=25.0,
**kw,
)
def _pool_kw(conn_params: ConnParams) -> dict:
return {
"host": conn_params.host,
"port": conn_params.port,
"user": conn_params.user,
"password": conn_params.password,
"database": conn_params.database,
"server": conn_params.server,
"connect_timeout": 15.0,
"read_timeout": 30.0,
}
# ---------------------------------------------------------------------------
# Error recovery — a failed statement must not poison the connection
# ---------------------------------------------------------------------------
FAILING_SQL = [
pytest.param("SELECT FROM WHERE", id="syntax"),
pytest.param("SELECT * FROM no_such_table_xyz", id="no_such_table"),
pytest.param("SELECT no_such_col FROM systables", id="no_such_column"),
pytest.param("SELECT no_such_function_xyz(1) FROM systables", id="bad_func"),
pytest.param("SELECT 1/0 FROM systables", id="div_zero"),
pytest.param("SELECT 'notanumber'::INT FROM systables", id="bad_cast"),
]
@pytest.mark.parametrize("sql", FAILING_SQL)
def test_connection_survives_failed_statement(
conn_params: ConnParams, sql: str
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
for _ in range(4): # repeat: a leak accumulates
with pytest.raises(informix_db.Error):
cur.execute(sql)
cur.fetchall()
cur.execute("SELECT FIRST 1 tabid FROM systables ORDER BY tabid")
assert cur.fetchone() is not None
@pytest.mark.parametrize("autocommit", [True, False])
def test_duplicate_key_does_not_kill_the_connection(
conn_params: ConnParams, autocommit: bool
) -> None:
"""The bug that started this file. A unique-constraint violation left
the prepared statement allocated server-side; the next statement then
failed with a nonsense error (-255 "Not in transaction" in autocommit,
-285 otherwise) and the connection never recovered."""
with _connect(conn_params, autocommit=autocommit) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_dup (k INT PRIMARY KEY)")
cur.execute("INSERT INTO t_dup VALUES (1)")
if not autocommit:
conn.commit()
for i in range(5):
with pytest.raises(informix_db.IntegrityError):
cur.execute("INSERT INTO t_dup VALUES (1)")
if not autocommit:
conn.rollback()
cur.execute("SELECT k FROM t_dup")
assert cur.fetchall() == [(1,)], f"broken after {i + 1} violations"
def test_failed_dml_with_params_releases_statement(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_nn (s VARCHAR(8) NOT NULL, k INT)")
for _ in range(4):
with pytest.raises(informix_db.Error):
cur.execute("INSERT INTO t_nn VALUES (?, ?)", (None, 1))
cur.execute("SELECT COUNT(*) FROM t_nn")
assert cur.fetchone() == (0,)
def test_new_cursor_works_after_error(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.Error):
cur.execute("SELECT * FROM no_such_table_xyz")
other = conn.cursor()
other.execute("SELECT FIRST 1 tabid FROM systables")
assert other.fetchone() is not None
other.close()
# ---------------------------------------------------------------------------
# Fetch batching — NFETCH is a byte budget, so batch edges move with width
# ---------------------------------------------------------------------------
@pytest.mark.parametrize("n", [0, 1, 2, 3, 63, 64, 65, 199, 200, 201, 512, 1025])
def test_row_count_exact_across_fetch_styles(
conn_params: ConnParams, n: int
) -> None:
"""Every fetch style must agree, at counts that straddle plausible
server batch boundaries. A batching off-by-one shows up as a wrong
count, a duplicate, or a dropped row."""
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_batch "
"(k INT, a VARCHAR(32), b INT, c VARCHAR(32), d INT)"
)
if n:
cur.executemany(
"INSERT INTO t_batch VALUES (?, ?, ?, ?, ?)",
[(i, f"a{i}", i * 2, f"c{i}", i * 3) for i in range(n)],
)
want = [(i,) for i in range(n)]
sql = "SELECT k FROM t_batch ORDER BY k"
cur.execute(sql)
assert cur.fetchall() == want
cur.execute(sql)
got = []
while (row := cur.fetchone()) is not None:
got.append(row)
assert got == want
for size in (1, 7, 100):
cur.execute(sql)
got = []
while chunk := cur.fetchmany(size):
got.extend(chunk)
assert got == want, f"fetchmany({size}) disagreed at n={n}"
cur.execute(sql)
assert list(cur) == want
# ---------------------------------------------------------------------------
# Cursor lifecycle
# ---------------------------------------------------------------------------
def test_abandoned_partial_fetches_do_not_leak(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_aband (k INT)")
cur.executemany(
"INSERT INTO t_aband VALUES (?)", [(i,) for i in range(200)]
)
for i in range(25):
c = conn.cursor()
c.execute("SELECT k FROM t_aband ORDER BY k")
c.fetchone()
if i % 2:
c.close()
else:
del c # rely on the finalizer
cur.execute("SELECT COUNT(*) FROM t_aband")
assert cur.fetchone() == (200,)
def test_re_execute_mid_fetch(conn_params: ConnParams) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_reex (k INT)")
cur.executemany(
"INSERT INTO t_reex VALUES (?)", [(i,) for i in range(100)]
)
for _ in range(10):
cur.execute("SELECT k FROM t_reex ORDER BY k")
cur.fetchone() # abandon mid-fetch
cur.execute("SELECT k FROM t_reex ORDER BY k")
assert len(cur.fetchall()) == 100
def test_interleaved_cursors_on_one_connection(
conn_params: ConnParams,
) -> None:
with _connect(conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_inter (k INT)")
cur.executemany(
"INSERT INTO t_inter VALUES (?)", [(i,) for i in range(50)]
)
a, b = conn.cursor(), conn.cursor()
a.execute("SELECT k FROM t_inter ORDER BY k")
b.execute("SELECT k FROM t_inter ORDER BY k DESC")
assert [a.fetchone() for _ in range(5)] == [(i,) for i in range(5)]
assert [b.fetchone() for _ in range(5)] == [
(i,) for i in range(49, 44, -1)
]
a.close()
b.close()
# ---------------------------------------------------------------------------
# Concurrency
# ---------------------------------------------------------------------------
def test_threads_sharing_one_connection_do_not_interleave(
conn_params: ConnParams,
) -> None:
"""One connection is one socket. Each thread reads back a value only
it supplied, so crossed wires fail even if nothing raises."""
failures: list[str] = []
lock = threading.Lock()
with _connect(conn_params, autocommit=True) as conn:
def worker(tid: int) -> None:
try:
for r in range(10):
cur = conn.cursor()
token = tid * 1000 + r
cur.execute(
"SELECT FIRST 1 ?::INT, ?::VARCHAR(16) FROM systables",
(token, f"t{tid}"),
)
row = cur.fetchone()
if row != (token, f"t{tid}"):
with lock:
failures.append(f"thread {tid}: got {row!r}")
return
cur.close()
except Exception as exc:
with lock:
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
for t in threads:
t.start()
for t in threads:
t.join()
assert not failures, failures
def test_pool_hands_out_clean_connections_under_load(
conn_params: ConnParams,
) -> None:
"""Borrowers that fail must still return a usable connection."""
failures: list[str] = []
lock = threading.Lock()
pool = informix_db.create_pool(
**_pool_kw(conn_params), min_size=2, max_size=4,
acquire_timeout=30.0, autocommit=True,
)
try:
def worker(tid: int) -> None:
try:
for r in range(8):
with pool.connection() as conn:
cur = conn.cursor()
if r % 3 == 0:
with contextlib.suppress(informix_db.Error):
cur.execute("SELECT * FROM no_such_tbl_xyz")
token = tid * 1000 + r
cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (token,)
)
got = cur.fetchone()
if got != (token,):
with lock:
failures.append(
f"thread {tid}: got {got!r}, want {token}"
)
return
cur.close()
except Exception as exc:
with lock:
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(6)]
for t in threads:
t.start()
for t in threads:
t.join()
finally:
with contextlib.suppress(Exception):
pool.close()
assert not failures, failures
def test_pool_does_not_leak_transactions_between_borrowers(
conn_params: ConnParams,
) -> None:
"""max_size=1 so both borrowers get the same underlying connection."""
pool = informix_db.create_pool(
**_pool_kw(conn_params), min_size=1, max_size=1,
acquire_timeout=30.0, autocommit=False,
)
try:
with pool.connection() as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_pool_txn")
cur.execute("CREATE TABLE t_pool_txn (k INT)")
conn.commit()
try:
for i in range(5):
with pool.connection() as conn: # dirties, never commits
conn.cursor().execute(
"INSERT INTO t_pool_txn VALUES (?)", (i,)
)
with pool.connection() as conn: # must not see it
cur = conn.cursor()
cur.execute("SELECT COUNT(*) FROM t_pool_txn")
assert cur.fetchone() == (0,), "transaction leaked"
conn.rollback()
finally:
with pool.connection() as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_pool_txn")
conn.commit()
finally:
with contextlib.suppress(Exception):
pool.close()
# ---------------------------------------------------------------------------
# Async
# ---------------------------------------------------------------------------
async def test_async_pool_survives_cancelled_acquires(
conn_params: ConnParams,
) -> None:
"""Cancelling a task blocked on acquire used to destroy that pool slot
permanently: ``asyncio.to_thread`` cannot interrupt its worker, so the
worker finished and handed back a connection nobody owned.
Two holders keep both connections checked out; three waiters block on
acquire and are cancelled there. Afterwards the pool must still serve.
"""
pool = await aio.create_pool(
**_pool_kw(conn_params), min_size=1, max_size=2, autocommit=True
)
try:
release = asyncio.Event()
async def holder(i: int) -> None:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (i,)
)
await cur.fetchone()
await release.wait()
async def waiter(i: int) -> tuple | None:
async with pool.connection() as conn:
cur = await conn.cursor()
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (i,)
)
return await cur.fetchone()
for cycle in range(2):
release.clear()
holders = [asyncio.create_task(holder(i)) for i in range(2)]
await asyncio.sleep(0.5) # both connections held
waiters = [asyncio.create_task(waiter(100 + i)) for i in range(3)]
await asyncio.sleep(0.5) # all blocked on acquire
for t in waiters:
t.cancel()
await asyncio.gather(*waiters, return_exceptions=True)
release.set()
await asyncio.gather(*holders, return_exceptions=True)
await asyncio.sleep(0.3) # orphan returns land
got = await asyncio.wait_for(waiter(200 + cycle), timeout=15)
assert got == (200 + cycle,), (
f"pool starved after cycle {cycle} — cancelled acquires leaked"
)
finally:
with contextlib.suppress(Exception):
await pool.close()
async def test_async_pool_concurrent_tasks_get_their_own_data(
conn_params: ConnParams,
) -> None:
pool = await aio.create_pool(
**_pool_kw(conn_params), min_size=2, max_size=4, autocommit=True
)
try:
async def worker(tid: int) -> None:
for r in range(6):
async with pool.connection() as conn:
cur = await conn.cursor()
token = tid * 1000 + r
await cur.execute(
"SELECT FIRST 1 ?::INT FROM systables", (token,)
)
assert await cur.fetchone() == (token,)
await asyncio.gather(*(worker(i) for i in range(6)))
finally:
with contextlib.suppress(Exception):
await pool.close()

View File

@ -0,0 +1,230 @@
"""End-of-row reconciliation, and the two bugs it found immediately.
Fourteen framing bugs reached users before this check existed. Every one
was the same defect a column consuming the wrong number of bytes and
not one raised at the point of the mistake. The wrong width produced a
plausible value and corrupted whatever came *next*, so the damage always
surfaced in a different column, row, or statement.
All fourteen were detectable for free. ``payload`` is a fully extracted
``bytes`` object of known length, so after decoding N columns the offset
must land exactly on the end. ``_row_not_consumed`` raises when it
doesn't, naming the column shape and the byte delta.
Turning the check on found two more bugs on its first run against the
existing suite:
* **Smart LOBs were read as a flat 72-byte field.** They use the UDT
envelope: 149 bytes populated (``[ind=0][len=144][144 hex chars]``),
5 bytes NULL. We consumed 72 and left 77 behind, so any LOB not in the
final column position corrupted everything after it. The 144 bytes are
the 72-byte locator *hex-encoded*, which means ``BlobLocator.raw`` had
never actually held a locator only the first half of the hex text.
Nobody noticed because ``read_blob_column`` resolves LOBs through the
server-side ``lotofile`` function and never uses the locator.
* **NULL composite UDTs skipped their length field**, byte-for-byte the
LVARCHAR NULL bug in the branch twelve lines away.
Both were the same ``[indicator][int32 length][data]`` envelope, written
out by hand a third and fourth time. ``_read_udt_envelope`` is now the
single implementation.
"""
from __future__ import annotations
import contextlib
import io
import struct
import pytest
import informix_db
from informix_db._protocol import IfxStreamReader, ProtocolError
from informix_db._resultset import (
ColumnInfo,
_read_udt_envelope,
parse_tuple_payload,
)
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The check itself — unit level, no server
# ---------------------------------------------------------------------------
def _tuple_pdu(body: bytes) -> IfxStreamReader:
"""Wrap a payload in the SQ_TUPLE framing parse_tuple_payload expects."""
return IfxStreamReader(
io.BytesIO(struct.pack("!h", 0) + struct.pack("!i", len(body)) + body)
)
def test_under_read_is_detected() -> None:
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
with pytest.raises(ProtocolError, match="under-read by 6"):
parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5) + b"LEFTOV"), cols)
def test_exact_consumption_passes() -> None:
cols = [ColumnInfo(name="k", type_code=2, raw_type_code=2, encoded_length=4)]
assert parse_tuple_payload(_tuple_pdu(struct.pack("!i", 5)), cols) == (5,)
def test_error_names_the_column_shape() -> None:
""""row decode failed" is not actionable. The message has to say which
shape and by how much, so the next report arrives with the answer."""
cols = [
ColumnInfo(name="ident", type_code=2, raw_type_code=2, encoded_length=4),
ColumnInfo(name="qty", type_code=2, raw_type_code=2, encoded_length=4),
]
body = struct.pack("!i", 1) + struct.pack("!i", 2) + b"UNCONSUMED"
with pytest.raises(ProtocolError) as exc:
parse_tuple_payload(_tuple_pdu(body), cols)
msg = str(exc.value)
assert "ident" in msg and "qty" in msg, "message must name the columns"
assert "tc=2" in msg
assert "consumed 8 of 18" in msg, "message must give the byte counts"
assert "under-read by 10" in msg
# ---------------------------------------------------------------------------
# The shared UDT envelope
# ---------------------------------------------------------------------------
def test_envelope_reads_length_even_when_null() -> None:
"""The length belongs to the envelope, not the value. Returning early
on the indicator is the bug that hit LVARCHAR and then composites."""
payload = bytes([1]) + struct.pack("!i", 0) + b"NEXT"
offset, body = _read_udt_envelope(payload, 0)
assert body is None
assert offset == 5, "NULL envelope must still consume its length field"
assert payload[offset:] == b"NEXT"
def test_envelope_reads_body_when_present() -> None:
payload = bytes([0]) + struct.pack("!i", 3) + b"abc" + b"NEXT"
offset, body = _read_udt_envelope(payload, 0)
assert body == b"abc"
assert offset == 8
assert payload[offset:] == b"NEXT"
def test_envelope_rejects_negative_length() -> None:
"""A negative length off the wire would rewind the offset and silently
re-decode earlier bytes as a later column."""
payload = bytes([0]) + struct.pack("!i", -4) + b"abc"
with pytest.raises(ProtocolError, match="negative length"):
_read_udt_envelope(payload, 0)
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
pytestmark_integration = pytest.mark.integration
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host, port=conn_params.port, user=conn_params.user,
password=conn_params.password, database=conn_params.database,
server=conn_params.server, connect_timeout=15.0, read_timeout=30.0,
autocommit=True,
)
@pytest.mark.integration
def test_null_composite_does_not_shift_next_column(
conn_params: ConnParams,
) -> None:
"""A NULL SET made the following VARCHAR return '' instead of its value."""
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_nullset")
cur.execute(
"CREATE TABLE t_nullset (a INT, s SET(INT NOT NULL), b VARCHAR(10))"
)
try:
cur.execute("INSERT INTO t_nullset VALUES (7, SET{1,2}, 'abc')")
cur.execute("INSERT INTO t_nullset VALUES (8, NULL, 'xyz')")
cur.execute("SELECT a, s, b FROM t_nullset ORDER BY a")
rows = cur.fetchall()
assert rows[0][0] == 7 and rows[0][2] == "abc"
assert rows[1] == (8, None, "xyz"), (
"NULL composite shifted the following column"
)
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_nullset")
@pytest.mark.integration
@pytest.mark.parametrize("populated", [False, True])
def test_lob_in_non_final_position(
conn_params: ConnParams, populated: bool
) -> None:
"""A LOB was read as 72 bytes when it occupies 149 (or 5 when NULL), so
anything selected after it was garbage."""
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_lobmid")
try:
cur.execute(
"CREATE TABLE t_lobmid (a INT, b BLOB, c INT, d VARCHAR(10))"
)
except informix_db.Error:
pytest.skip("no sbspace configured; see make ifx-spaces")
try:
if populated:
cur.write_blob_column(
"INSERT INTO t_lobmid VALUES (?, BLOB_PLACEHOLDER, ?, ?)",
b"hello", (1, 222222, "tail"),
)
else:
cur.execute(
"INSERT INTO t_lobmid VALUES (1, NULL, 222222, 'tail')"
)
cur.execute("SELECT a, b, c, d FROM t_lobmid")
a, b, c, d = cur.fetchone()
assert (a, c, d) == (1, 222222, "tail"), (
"columns after the LOB were shifted"
)
if populated:
assert isinstance(b, informix_db.BlobLocator)
assert len(b.raw) == 72, (
"locator must be the decoded 72 bytes, not hex text"
)
else:
assert b is None
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_lobmid")
@pytest.mark.integration
def test_wide_mixed_row_reconciles(conn_params: ConnParams) -> None:
"""Belt and braces: a row using most of the tricky types must consume
its payload exactly. The check runs on every fetch, so merely getting
a row back proves reconciliation held."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_recon ("
" a INT8 NOT NULL, k LVARCHAR(512), n NCHAR(8), v VARCHAR(32),"
" d DECIMAL(16), b BOOLEAN, t DATETIME YEAR TO FRACTION(5),"
" tail INT)"
)
cur.execute(
"INSERT INTO t_recon VALUES (?, ?, ?, ?, ?, ?, CURRENT, ?)",
(1, "PackageRoot", "nch", "v", None, True, 424242),
)
cur.execute(
"SELECT a, k, n, v, d, b, t, tail FROM t_recon"
)
row = cur.fetchone()
assert row[0] == 1
assert row[-1] == 424242

435
tests/test_rows.py Normal file
View File

@ -0,0 +1,435 @@
"""Named row access, and the cost of it.
A field request: `row[11]` tells a reader nothing on a wide projection,
and `pyodbc` / `mssql-python` both hand back rows that answer to
position, column name, and attribute at once. `row_factory=Row` gives
the same three.
It is opt-in. Defaulting it on would tax the thing this driver is
measured against, since supporting `row["name"]` means `__getitem__`
becomes a Python method rather than C-level tuple indexing, which costs
roughly 39 ns on every subscript. Users who want readable column access
in application code should pay that; a bulk export that never looks at a
column by name should not.
`Row` subclasses `tuple`, so `row == (1, "x")` still holds and nothing
that already worked stops working. That constraint drove the design: the
existing suite compares fetched rows against plain tuples in hundreds of
places, and so does everybody's code.
"""
from __future__ import annotations
import contextlib
import pickle
import pytest
import informix_db
from informix_db.rows import _RESERVED, Row, make_row_class
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The type itself
# ---------------------------------------------------------------------------
def _row(fields, values):
return make_row_class(tuple(fields))(values)
def test_three_ways_to_reach_a_column() -> None:
row = _row(("tabid", "tabname"), (1, "systables"))
assert row[0] == 1
assert row["tabname"] == "systables"
assert row.tabname == "systables"
def test_still_equal_to_a_plain_tuple() -> None:
"""The compatibility constraint. Existing code and the existing test
suite compare fetched rows against tuples everywhere."""
row = _row(("a", "b"), (1, "x"))
plain = (1, "x")
assert row == plain
# Both directions: tuple.__eq__ on the left has to accept a subclass
# on the right, or `expected == fetched` assertions break.
assert plain == row
assert list(row) == [1, "x"]
assert len(row) == 2
a, b = row
assert (a, b) == (1, "x")
assert row in [(1, "x")]
def test_slice_degrades_to_a_plain_tuple() -> None:
"""A slice has no meaningful column mapping, so it should not pretend
to be a Row."""
row = _row(("a", "b", "c"), (1, 2, 3))
assert row[0:2] == (1, 2)
assert type(row[0:2]) is tuple
def test_negative_index_still_works() -> None:
assert _row(("a", "b"), (1, 2))[-1] == 2
def test_keys_and_asdict() -> None:
row = _row(("a", "b"), (1, "x"))
assert row.keys() == ("a", "b")
assert row._asdict() == {"a": 1, "b": "x"}
def test_repr_names_the_columns() -> None:
assert repr(_row(("a", "b"), (1, "x"))) == "Row(a=1, b='x')"
def test_pickles() -> None:
"""The per-shape class is built at runtime, so it cannot be pickled by
reference. Multiprocessing users would hit that immediately."""
row = _row(("a", "b"), (1, "x"))
restored = pickle.loads(pickle.dumps(row))
assert restored == (1, "x")
assert restored.b == "x"
def test_missing_column_says_what_is_there() -> None:
row = _row(("tabid", "tabname"), (1, "x"))
with pytest.raises(KeyError, match="tabid"):
_ = row["nope"]
with pytest.raises(AttributeError, match="tabid"):
_ = row.nope
def test_class_is_cached_per_shape() -> None:
"""A type() call per execute() would land on every small query."""
assert make_row_class(("a", "b")) is make_row_class(("a", "b"))
assert make_row_class(("a", "b")) is not make_row_class(("a", "c"))
def test_duplicate_names_resolve_to_the_first() -> None:
"""``SELECT tabid, tabid`` is legal and both columns are named
``tabid``. pyodbc gives the first; so do we."""
row = _row(("tabid", "tabid"), (1, 2))
assert row["tabid"] == 1
assert row.tabid == 1
assert row[1] == 2, "positional access must still reach the second"
@pytest.mark.parametrize("name", sorted(_RESERVED))
def test_a_column_always_beats_a_method_of_the_same_name(name: str) -> None:
"""Parametrized over the *computed* reserved set rather than a
hand-written list, because the hand-written list is what missed
``keys``, ``_asdict`` and ``_fields`` on the first attempt. Adding a
method to Row later cannot silently reopen the hole: this test grows
with it.
Without the guard these return a bound method, which is the exact
failure shape this driver spent a fortnight removing from its
decoders: a plausible-looking wrong answer, in silence."""
row = _row((name, "other"), (7, 9))
assert getattr(row, name) == 7
assert row[name] == 7
def test_shadowed_methods_stay_reachable_through_the_base_class() -> None:
row = _row(("count", "keys", "_asdict"), (1, 2, 3))
assert tuple.count(row, 1) == 1
assert Row.keys(row) == ("count", "keys", "_asdict")
assert Row._asdict(row) == {"count": 1, "keys": 2, "_asdict": 3}
def test_shadowing_fields_does_not_break_repr_or_asdict() -> None:
"""The machinery reads its own field list, so if a column named
``_fields`` shadowed it, repr and _asdict would report the column
value instead of the names. Mangled attributes keep them separate."""
row = _row(("_fields", "x"), ("not the names", 2))
assert row._fields == "not the names"
assert Row.keys(row) == ("_fields", "x")
assert repr(row) == "Row(_fields='not the names', x=2)"
def test_non_identifier_names_are_subscript_only() -> None:
"""Informix names expression columns things like ``(count(*))``,
which cannot be an attribute."""
row = _row(("(count(*))", "ok"), (5, 1))
assert row["(count(*))"] == 5
assert row.ok == 1
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
**kw,
)
@pytest.mark.integration
def test_default_is_still_a_plain_tuple(conn_params: ConnParams) -> None:
"""The opt-in has to be genuinely opt-in. Anyone who does not ask for
Row should not pay for it or see any behaviour change."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 1 tabid, tabname FROM systables")
row = cur.fetchone()
assert type(row) is tuple
with pytest.raises(TypeError):
_ = row["tabname"]
@pytest.mark.integration
def test_row_factory_on_the_connection(conn_params: ConnParams) -> None:
"""One line at connect time, then every cursor from it, which is what
'without any additional coding' has to mean in practice."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
cur.execute(
"SELECT FIRST 1 tabid, tabname FROM systables ORDER BY tabid"
)
row = cur.fetchone()
assert row[0] == row["tabid"] == row.tabid
assert row[1] == row["tabname"] == row.tabname
@pytest.mark.integration
def test_every_fetch_path_returns_rows(conn_params: ConnParams) -> None:
"""fetchone, fetchmany, fetchall and iteration each return rows by a
different route through the cursor."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
sql = "SELECT FIRST 4 tabid, tabname FROM systables ORDER BY tabid"
cur = conn.cursor()
cur.execute(sql)
assert cur.fetchone().tabname is not None
cur.execute(sql)
assert all(r.tabname is not None for r in cur.fetchmany(2))
cur.execute(sql)
assert all(r.tabname is not None for r in cur.fetchall())
cur.execute(sql)
assert all(r.tabname is not None for r in cur)
@pytest.mark.integration
def test_scrollable_cursor_returns_rows(conn_params: ConnParams) -> None:
"""Scrollable cursors return each row straight from the wire rather
than from the materialized list, so they are a separate path."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor(scrollable=True)
cur.execute("SELECT tabid, tabname FROM systables ORDER BY tabid")
assert cur.fetch_first().tabname is not None
assert cur.fetch_absolute(1).tabname is not None
cur.close()
@pytest.mark.integration
def test_per_cursor_override(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
named = conn.cursor()
named.row_factory = informix_db.Row
plain = conn.cursor()
sql = "SELECT FIRST 1 tabid FROM systables"
named.execute(sql)
plain.execute(sql)
assert isinstance(named.fetchone(), Row)
assert type(plain.fetchone()) is tuple
@pytest.mark.integration
def test_informix_lowercases_so_the_obvious_name_works(
conn_params: ConnParams,
) -> None:
"""Informix folds unquoted identifiers, which is why the .lower() in
the workaround people write by hand is a no-op."""
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_rows (Config_Key INT, Cnt INT)")
cur.execute("INSERT INTO t_rows VALUES (1, 2)")
cur.execute("SELECT Config_Key, Cnt FROM t_rows")
row = cur.fetchone()
assert row.config_key == 1
assert row.cnt == 2
@pytest.mark.integration
def test_pool_forwards_the_factory(conn_params: ConnParams) -> None:
pool = informix_db.create_pool(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
autocommit=True,
row_factory=informix_db.Row,
min_size=1,
max_size=2,
)
try:
with pool.connection() as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone().tabid is not None
finally:
pool.close()
def test_zero_column_row() -> None:
row = make_row_class(())(())
assert row == ()
assert row.keys() == ()
assert repr(row) == "Row()"
def test_wide_row() -> None:
"""500 columns: the name map is a dict, so this should be flat, but
it is the shape most likely to expose an off-by-one."""
fields = tuple(f"c{i}" for i in range(500))
row = make_row_class(fields)(tuple(range(500)))
assert row["c499"] == row.c499 == row[499] == 499
assert row["c0"] == row[0] == 0
def test_names_that_cannot_be_attributes_are_subscript_only() -> None:
row = _row(("col with space", "1leading_digit", ""), (1, 2, 3))
assert row["col with space"] == 1
assert row["1leading_digit"] == 2
assert row[""] == 3
def test_unicode_column_name() -> None:
row = _row(("café", "x"), (1, 2))
assert row["café"] == 1
assert row.café == 1
def test_dunder_named_column_is_subscript_only() -> None:
"""A dunder cannot be shadowed without breaking the object protocol,
so the column is reachable by subscript and the attribute keeps its
ordinary meaning. No real schema should notice."""
row = _row(("__class__", "ok"), (1, 2))
assert row["__class__"] == 1
assert row.__class__.__name__ == "Row"
def test_row_outlives_eviction_of_its_class_from_the_cache() -> None:
"""The class cache is bounded. A row already handed to the caller
holds its class alive, so eviction must not affect it."""
row = make_row_class(("z1", "z2"))((9, 8))
for i in range(300):
make_row_class((f"evict{i}", "b"))
assert (row.z1, row["z2"], row[0]) == (9, 8, 9)
def test_class_cache_is_bounded() -> None:
for i in range(400):
make_row_class((f"bounded{i}", "b"))
assert make_row_class.cache_info().currsize <= 256
# ---------------------------------------------------------------------------
# Row must not change a single value
# ---------------------------------------------------------------------------
_WIDE_DDL = """CREATE TABLE t_rowdiff (
c_int INT, c_small SMALLINT, c_big BIGINT, c_int8 INT8, c_serial SERIAL,
c_float FLOAT, c_smallfloat SMALLFLOAT, c_dec DECIMAL(16,4),
c_decu DECIMAL(16), c_money MONEY(12,2), c_char CHAR(10),
c_vchar VARCHAR(40), c_nchar NCHAR(8), c_lvar LVARCHAR(200), c_date DATE,
c_dt DATETIME YEAR TO FRACTION(5), c_ivl INTERVAL YEAR TO MONTH,
c_bool BOOLEAN, c_set SET(INT NOT NULL), c_tail INT)"""
_WIDE_COLS = (
"c_int,c_small,c_big,c_int8,c_serial,c_float,c_smallfloat,c_dec,c_decu,"
"c_money,c_char,c_vchar,c_nchar,c_lvar,c_date,c_dt,c_ivl,c_bool,c_set,"
"c_tail"
)
@pytest.mark.integration
def test_rows_are_value_identical_to_tuples(conn_params: ConnParams) -> None:
"""The strongest statement available: across every awkward type, with
a fully populated row and a fully NULL one, wrapping must change
nothing. If it does, Row is not a presentation layer, it is a bug."""
sql = f"SELECT {_WIDE_COLS} FROM t_rowdiff ORDER BY c_tail"
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_rowdiff")
cur.execute(_WIDE_DDL)
try:
cur.execute(
"INSERT INTO t_rowdiff VALUES (1,2,3,4,0,1.5,2.5,12.34,99,"
"9.99,'ch','vc','nc','lv',TODAY,CURRENT,"
"INTERVAL(1-2) YEAR TO MONTH,'t',SET{1,2},77)"
)
cur.execute(
"INSERT INTO t_rowdiff VALUES (NULL,NULL,NULL,NULL,0,NULL,"
"NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,"
"NULL,NULL,88)"
)
cur.execute(sql)
plain = cur.fetchall()
with _connect(conn_params, row_factory=informix_db.Row) as named_conn:
named = named_conn.cursor()
named.execute(sql)
rows = named.fetchall()
assert rows == plain, "wrapping changed a value"
names = _WIDE_COLS.split(",")
first = rows[0]
for i, name in enumerate(names):
assert first[i] == first[name] == getattr(first, name), name
assert rows[1]["c_int"] is None
assert rows[1].c_tail == 88
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_rowdiff")
@pytest.mark.integration
def test_threads_share_one_class_per_shape(conn_params: ConnParams) -> None:
"""The cache is process-wide and the pool hands connections to many
threads, so two threads running the same query must land on the same
class rather than racing to build competing ones."""
import threading
seen: list[type] = []
errors: list[Exception] = []
def worker() -> None:
try:
with _connect(conn_params, row_factory=informix_db.Row) as conn:
cur = conn.cursor()
for _ in range(5):
cur.execute(
"SELECT FIRST 1 tabid, tabname FROM systables"
)
row = cur.fetchone()
assert row.tabname == row["tabname"] == row[1]
seen.append(type(row))
except Exception as exc:
errors.append(exc)
threads = [threading.Thread(target=worker) for _ in range(6)]
for t in threads:
t.start()
for t in threads:
t.join(60)
assert not errors, errors[:2]
assert len({id(c) for c in seen}) == 1, "same shape built more than once"

172
tests/test_socket_reads.py Normal file
View File

@ -0,0 +1,172 @@
"""Two readers, one stream, and a length field taken on trust.
``IfxSocket`` owns a read-ahead buffer that ``BufferedSocketReader``
fills and drains. But ``Connection._drain_to_eot``, ``_raise_sq_err``
and the login path bypass that reader and call ``IfxSocket.read_exact``
directly, which recv'd from the socket without ever looking at the
buffer. Bytes sitting in the buffer would simply be skipped, and skipped
bytes in a length-framed protocol don't announce themselves — the next
read lands mid-field and every read after it is wrong.
Nothing triggers it today. The server sends one response per request, so
recv returns exactly that response and the buffered reader consumes all
of it before control returns to a direct read. That is a property of the
traffic, not of the code, and the buffer is connection-scoped precisely
so read-ahead *can* cross response boundaries pipelined executemany
already puts several responses in flight. A latent desync waiting on a
timing change is not a good thing to leave in a wire protocol.
Separately, ``fill_recv_buf`` took its byte count on trust, and that
count is almost always a length field straight off the wire. A corrupt
or desynced stream turned into an allocation of whatever the field
happened to say: a garbage ``0x7FFFFFFF`` reads as a 2 GB request, and
the fill loop sits in recv until the read timeout while the buffer
grows. The limit turns that into an error that names the number, which
is the actual diagnostic a length that absurd means framing was
already lost upstream.
"""
from __future__ import annotations
import pytest
from informix_db._protocol import BufferedSocketReader, ProtocolError
from informix_db._socket import MAX_READ_BYTES, IfxSocket
class _FakeSocket:
"""Stands in for the raw socket. Records what recv actually asked for."""
def __init__(self, data: bytes = b"") -> None:
self.data = data
self.pos = 0
self.recv_calls: list[int] = []
def recv(self, n: int) -> bytes:
self.recv_calls.append(n)
chunk = self.data[self.pos : self.pos + n]
self.pos += len(chunk)
return chunk
def close(self) -> None:
# The EOF path force-closes; a stand-in has to survive that.
pass
def _socket_with(buffered: bytes, on_wire: bytes = b"") -> IfxSocket:
"""An IfxSocket with ``buffered`` already read ahead into _recv_buf."""
sock = IfxSocket.__new__(IfxSocket)
sock._sock = _FakeSocket(on_wire)
sock._recv_buf = bytearray(buffered)
sock._recv_pos = 0
sock._recv_size = 65536
sock._read_timeout = None
return sock
# ---------------------------------------------------------------------------
# read_exact must not step over the buffer
# ---------------------------------------------------------------------------
def test_read_exact_consumes_the_buffer_first() -> None:
sock = _socket_with(b"BUFFERED", on_wire=b"SOCKET")
assert sock.read_exact(8) == b"BUFFERED"
assert sock._sock.recv_calls == [], "must not touch the socket at all"
assert sock._recv_pos == 8
def test_read_exact_spans_buffer_then_socket() -> None:
"""The interesting case: a read that starts in the buffer and
finishes on the wire. Getting this wrong reorders the stream."""
sock = _socket_with(b"HEAD", on_wire=b"TAIL")
assert sock.read_exact(8) == b"HEADTAIL"
assert sock._sock.recv_calls == [4], "only the shortfall comes from recv"
def test_read_exact_respects_a_partly_consumed_buffer() -> None:
sock = _socket_with(b"XXABCD")
sock._recv_pos = 2 # first two bytes already decoded
assert sock.read_exact(4) == b"ABCD"
assert sock._sock.recv_calls == []
def test_read_exact_of_zero_is_empty() -> None:
sock = _socket_with(b"DATA")
assert sock.read_exact(0) == b""
assert sock.read_exact(-5) == b"", "a negative count must not rewind"
assert sock._recv_pos == 0
def test_short_read_error_reports_the_original_request() -> None:
"""The message counts bytes; taking some from the buffer must not make
it lie about how many were asked for."""
from informix_db.exceptions import OperationalError
sock = _socket_with(b"AB", on_wire=b"") # 2 buffered, nothing on the wire
with pytest.raises(OperationalError, match="wanted 10 bytes"):
sock.read_exact(10)
def test_buffered_reader_and_direct_read_agree_on_one_stream() -> None:
"""End to end: a BufferedSocketReader over-reads, then a direct
read_exact picks up exactly where it left off."""
sock = _socket_with(b"", on_wire=b"\x00\x2aREST-OF-THE-STREAM")
reader = BufferedSocketReader(sock)
assert reader.read_short() == 42
assert len(sock._recv_buf) - sock._recv_pos > 0, (
"precondition: the reader must have over-read for this to mean "
"anything"
)
assert sock.read_exact(18) == b"REST-OF-THE-STREAM"
# ---------------------------------------------------------------------------
# fill_recv_buf must not believe an arbitrary length
# ---------------------------------------------------------------------------
def test_absurd_length_is_refused_not_allocated() -> None:
sock = _socket_with(b"", on_wire=b"")
with pytest.raises(ProtocolError, match="refusing to read"):
sock.fill_recv_buf(MAX_READ_BYTES + 1)
assert sock._sock.recv_calls == [], "must refuse before any recv"
def test_refusal_names_the_knob() -> None:
"""The error has to be actionable in both directions: framing is lost,
or the value genuinely is that big and the limit needs raising."""
sock = _socket_with(b"", on_wire=b"")
with pytest.raises(ProtocolError) as exc:
sock.fill_recv_buf(2**31 - 1)
message = str(exc.value)
assert "2147483647" in message
assert "IFX_MAX_READ_BYTES" in message
def test_a_normal_length_is_unaffected() -> None:
sock = _socket_with(b"", on_wire=b"x" * 100)
sock.fill_recv_buf(100)
assert len(sock._recv_buf) - sock._recv_pos >= 100
# ---------------------------------------------------------------------------
# skip
# ---------------------------------------------------------------------------
def test_buffered_skip_does_not_rewind_on_a_negative_count() -> None:
"""The base reader's skip delegates to read_exact, which guards. This
one advances the cursor arithmetically, so an unguarded negative count
re-decodes bytes already consumed as if they were the next field."""
sock = _socket_with(b"ABCDEFGH")
sock._recv_pos = 4
BufferedSocketReader(sock).skip(-4)
assert sock._recv_pos == 4, "skip must never move the cursor backwards"
def test_buffered_skip_advances_normally() -> None:
sock = _socket_with(b"ABCDEFGH")
reader = BufferedSocketReader(sock)
reader.skip(4)
assert reader.read_exact(4) == b"EFGH"

View File

@ -0,0 +1,216 @@
"""Deciding whether a statement needs a cursor, by asking rather than guessing.
The driver chose between "open a cursor and fetch" and "execute and
release" by checking whether the first word of the SQL was ``SELECT``.
That gets five ordinary forms wrong a leading comment in any of the
three Informix flavours, a parenthesized select, a parenthesized UNION,
and a CTE. All five are perfectly good queries, and all five failed with
``-260 Cursor name already in use``, an error that describes neither the
cause nor anything the caller did. It says "cursor" because the driver
sent SQ_EXECUTE where the server was waiting to open one.
The server had been telling us the answer the whole time.
``statement_type`` is the first field of the DESCRIBE response, and
``parse_describe`` has always parsed it into the metadata dict, where
nothing read it. Every SELECT form above reports 2.
The comment that justified the heuristic said ``nfields`` couldn't
distinguish these cases, because ``INSERT INTO t VALUES (?)`` also
describes a column. That much was true and it argued for the wrong
conclusion, because ``statement_type`` distinguishes them exactly. That
INSERT reports 6.
The predicate is now JDBC's ``IfxSqli.isResultSet``: type 2, or type 56
(``EXECUTE PROCEDURE``/``FUNCTION``) with at least one column, since a
routine may or may not return rows. That last clause fixes
``EXECUTE FUNCTION`` as a side effect it used to run down the DML path
and discard its return value.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _produces_result_set
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The predicate
# ---------------------------------------------------------------------------
@pytest.mark.parametrize(
("statement_type", "ncolumns", "expected", "why"),
[
(2, 1, True, "SELECT"),
(2, 0, True, "SELECT describing no columns is still a SELECT"),
(6, 1, False, "INSERT ... VALUES (?) describes a column but is DML"),
(6, 0, False, "INSERT with literals"),
(32, 0, False, "DELETE"),
(33, 0, False, "UPDATE"),
(45, 0, False, "CREATE"),
(56, 0, False, "EXECUTE PROCEDURE returning nothing"),
(56, 2, True, "EXECUTE FUNCTION returning rows"),
(0, 0, False, "unknown type defaults to the non-cursor path"),
],
)
def test_result_set_predicate(
statement_type: int, ncolumns: int, expected: bool, why: str
) -> None:
assert _produces_result_set(statement_type, ncolumns) is expected, why
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
)
_ONE = "SELECT FIRST 1 tabid FROM systables"
@pytest.mark.integration
@pytest.mark.parametrize(
("label", "sql"),
[
pytest.param("plain", _ONE, id="plain"),
pytest.param("lowercase", _ONE.lower(), id="lowercase"),
pytest.param("leading-whitespace", f" \n\t {_ONE}", id="whitespace"),
pytest.param("line-comment", f"-- pick one\n{_ONE}", id="line-comment"),
pytest.param("block-comment", f"/* pick one */ {_ONE}", id="block-comment"),
pytest.param("brace-comment", f"{{ pick one }} {_ONE}", id="brace-comment"),
pytest.param(
"cte",
"WITH c AS (SELECT tabid FROM systables) "
"SELECT FIRST 1 tabid FROM c",
id="cte",
),
pytest.param("parenthesized", f"({_ONE})", id="parenthesized"),
pytest.param(
"union", f"{_ONE} UNION SELECT 99 FROM systables", id="union"
),
pytest.param(
"parenthesized-union",
f"({_ONE}) UNION (SELECT 99 FROM systables)",
id="paren-union",
),
],
)
def test_every_select_form_opens_a_cursor(
conn_params: ConnParams, label: str, sql: str
) -> None:
"""The five non-``plain`` forms below the whitespace case all failed
with -260 under the first-word heuristic."""
with _connect(conn_params) as conn:
cur = conn.cursor()
try:
cur.execute(sql)
except informix_db.ProgrammingError as exc:
# Informix 12.10 has no CTEs and rejects WITH at offset 1.
# A syntax error is the server declining the grammar, which
# is a different thing from the driver routing it wrongly —
# that produced -260, not -201.
if getattr(exc, "sqlcode", None) == -201:
pytest.skip(f"server does not support this syntax: {label}")
raise
rows = cur.fetchall()
assert rows, f"{label}: expected rows, got none"
assert cur.description is not None
@pytest.mark.integration
def test_parameterized_insert_is_not_mistaken_for_a_query(
conn_params: ConnParams,
) -> None:
"""The case the old comment worried about, and the reason it kept the
heuristic: this DESCRIBEs a column. ``statement_type`` says 6."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_cls (k INT)")
cur.execute("INSERT INTO t_cls VALUES (?)", (7,))
assert cur.rowcount == 1
cur.execute("SELECT k FROM t_cls")
assert cur.fetchall() == [(7,)]
@pytest.mark.integration
def test_execute_function_returns_its_value(conn_params: ConnParams) -> None:
"""Type 56 with columns. Under the first-word heuristic this ran down
the DML path and the return value was discarded."""
with _connect(conn_params) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP FUNCTION ifxdrv_dbl")
cur.execute(
"CREATE FUNCTION ifxdrv_dbl(n INT) RETURNING INT; "
"RETURN n * 2; END FUNCTION"
)
try:
cur.execute("EXECUTE FUNCTION ifxdrv_dbl(21)")
assert cur.fetchall() == [(42,)]
finally:
with contextlib.suppress(Exception):
cur.execute("DROP FUNCTION ifxdrv_dbl")
@pytest.mark.integration
def test_dml_still_takes_the_non_cursor_path(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_cls2 (k INT)")
cur.execute("INSERT INTO t_cls2 VALUES (1)")
cur.execute("UPDATE t_cls2 SET k = 2")
assert cur.rowcount == 1
cur.execute("DELETE FROM t_cls2")
assert cur.rowcount == 1
assert cur.description is None
@pytest.mark.integration
def test_executemany_refuses_a_query_the_first_word_missed(
conn_params: ConnParams,
) -> None:
"""The pre-flight check shares the heuristic's blind spots, so a
comment-prefixed SELECT reaches PREPARE. The post-DESCRIBE check
catches it, and the connection stays usable the refusal releases
the statement. (A leading comment rather than a CTE, so this also
runs on 12.10, which has no CTEs.)"""
with _connect(conn_params) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.NotSupportedError):
cur.executemany(
"/* batched? no */ SELECT FIRST 1 tabid FROM systables "
"WHERE tabid <> ?",
[(1,), (2,)],
)
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, "refusal leaked the statement"
@pytest.mark.integration
def test_executemany_still_refuses_a_plain_select(
conn_params: ConnParams,
) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
with pytest.raises(informix_db.NotSupportedError):
cur.executemany(
"SELECT FIRST 1 tabid FROM systables WHERE tabid <> ?",
[(1,), (2,)],
)

View File

@ -0,0 +1,189 @@
"""Every door out of a statement has to release it on the way through.
A statement that fails is still allocated server-side. Skipping the
RELEASE bricks the connection: the next PREPARE collides with the leaked
one, and every subsequent call returns a nonsense error whose offset
points back at the *failed* SQL rather than the new statement.
There are six exits from the execute paths, and the guard was added to
them one at a time, each after a user hit it:
1. ``_execute_dml`` drain
2. ``_execute_dml_with_params`` build
3. ``_execute_dml_with_params`` drain
4. ``executemany`` pipeline build/send
5. ``_execute_select_with_params`` build
6. ``_execute_select`` fetch loop
Two were still open, and both are ordinary to reach:
* **The parameterized-SELECT bind drain.** The build was guarded and the
drain was not. Passing a string where the column is an INT gets a
clean encode, so the rejection comes from the *server* (-1213, -415)
during the bind drain past the guard. A wrong-typed parameter is
about as common as application mistakes get.
* **The scrollable-cursor open.** No guard at all, and the worst place
to lack one: the GC-time finalizer is armed on the line *after* the
drain, so a failure there left the statement allocated with no
fallback of any kind.
The guard is now one helper rather than six hand-rolled copies, which
also fixed a defect in the copies that had a cursor to close: CLOSE and
RELEASE shared a single ``contextlib.suppress`` block, so a CLOSE that
raised skipped the RELEASE losing the half that actually matters.
"""
from __future__ import annotations
import pytest
import informix_db
from informix_db.cursors import _RELEASE_PDU, Cursor
from tests.conftest import ConnParams
# ---------------------------------------------------------------------------
# The helper — no server needed
# ---------------------------------------------------------------------------
class _FakeConn:
def __init__(self) -> None:
self.sent: list[bytes] = []
def _send_pdu(self, pdu: bytes) -> None:
self.sent.append(pdu)
class _FakeCursor:
"""Duck-types the four attributes ``_release_after_failure`` touches."""
def __init__(self, *, close_raises: bool = False) -> None:
self._conn = _FakeConn()
self._close_raises = close_raises
def _build_close_pdu(self) -> bytes:
return b"CLOSE"
def _build_release_pdu(self) -> bytes:
return _RELEASE_PDU
def _drain_to_eot(self) -> None:
if self._close_raises and self._conn.sent[-1] == b"CLOSE":
raise OSError("wire went away mid-close")
def test_release_is_sent_even_when_close_fails() -> None:
"""The regression in the hand-rolled copies. CLOSE and RELEASE shared
one suppress block, so a failing CLOSE swallowed the RELEASE and a
lost cursor handle is a nuisance while a lost statement breaks the
next call."""
cur = _FakeCursor(close_raises=True)
Cursor._release_after_failure(cur, close_cursor=True)
assert _RELEASE_PDU in cur._conn.sent, (
"a CLOSE that raises must not prevent the RELEASE"
)
def test_close_is_skipped_when_no_cursor_was_opened() -> None:
cur = _FakeCursor()
Cursor._release_after_failure(cur)
assert cur._conn.sent == [_RELEASE_PDU]
def test_cleanup_never_propagates() -> None:
"""Cleanup runs inside an ``except``. If it raises, it replaces the
real SQL error with a secondary failure from the cleanup path, which
is strictly worse for the caller."""
class _Hostile(_FakeCursor):
def _drain_to_eot(self) -> None:
raise OSError("connection reset")
Cursor._release_after_failure(_Hostile(), close_cursor=True)
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
def _connect(conn_params: ConnParams) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
# Bounded: a leaked statement used to manifest as a hang.
read_timeout=25.0,
autocommit=True,
)
@pytest.mark.integration
@pytest.mark.parametrize(
("sql", "params"),
[
pytest.param(
"SELECT tabid FROM systables WHERE tabid = ?",
("not-an-int",),
id="string-for-int",
),
pytest.param(
"SELECT ?::INT FROM systables WHERE tabid = 1",
(2**40,),
id="out-of-range-int",
),
],
)
def test_server_rejected_bind_releases_statement(
conn_params: ConnParams, sql: str, params: tuple
) -> None:
"""A parameterized SELECT whose bind the *server* rejects. The value
encodes cleanly, so this lands past the build guard and in the drain
that had none. Repeated because a leak only shows on the call after
it the first failure looks fine on its own."""
with _connect(conn_params) as conn:
cur = conn.cursor()
for i in range(4):
with pytest.raises(informix_db.Error):
cur.execute(sql, params)
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, f"broken after {i + 1} binds"
@pytest.mark.integration
def test_failed_scroll_open_releases_statement(conn_params: ConnParams) -> None:
"""FOR UPDATE prepares cleanly and fails at OPEN with -526, which is
exactly the branch that had no guard."""
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
for i in range(4):
with pytest.raises(informix_db.Error):
scroll.execute("SELECT tabid FROM systables FOR UPDATE")
assert not scroll._server_cursor_open, (
"a failed open must not leave the cursor marked live"
)
other = conn.cursor()
other.execute("SELECT FIRST 1 tabid FROM systables")
assert other.fetchone() is not None, f"broken after {i + 1} opens"
other.close()
scroll.close()
@pytest.mark.integration
def test_scrollable_cursor_still_works_after_a_failed_open(
conn_params: ConnParams,
) -> None:
"""The same cursor object must be reusable — a failed open is an
ordinary error, not a terminal state for the cursor."""
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
with pytest.raises(informix_db.Error):
scroll.execute("SELECT tabid FROM systables FOR UPDATE")
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
assert scroll.fetch_first() is not None
scroll.close()

View File

@ -0,0 +1,355 @@
"""One statement per session, and what happens when we forget that.
SQLI gives a session a single statement slot. ``SQ_CLOSE``,
``SQ_RELEASE`` and ``SQ_SFETCH`` all act on whatever statement is
current none of them names one. Ordinary use never notices, because a
non-scrollable cursor materializes its rows and releases the statement
before returning, so the slot is free again by the time anyone looks.
A scrollable cursor is the exception: it holds the slot open on purpose.
Two things then went wrong, and neither said so.
**Another statement on the same connection.** The server returns ``-285``
for the new statement and *also* destroys the scrollable cursor its
next fetch comes back ``-267`` "the transaction has been rolled back,
all locks released". Two unattributable errors from code that reads as
completely ordinary: iterate a large result set, run a lookup partway
through. It is now a ``ProgrammingError`` that says what happened.
**The deferred-cleanup queue drained at the wrong moment.** A cursor
finalizer that can't get the wire lock hands its CLOSE/RELEASE to a
queue for the next operation to flush. That queue was flushed before
*every* PDU. But a finalizer enqueues precisely because another thread
holds the lock, i.e. is mid-statement so the flush landed inside that
thread's own statement and released it. The victim saw ``-208`` when it
happened before the first fetch, ``-267`` between fetch batches. The
flush now happens only at a statement boundary, where the queued CLOSE
addresses the orphan it was meant for.
A stale queue entry was fatal too: the finalizer enqueues, the cursor
then gets closed properly, and the leftover CLOSE draws ``-267`` an
``OperationalError``, which is in ``WIRE_ERRORS``, which force-closed a
perfectly healthy connection. Server-reported errors are now told apart
from wire failures by the presence of a ``sqlcode``.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _CLOSE_PDU, _RELEASE_PDU
from tests.conftest import ConnParams
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
autocommit=True,
**kw,
)
# ---------------------------------------------------------------------------
# The conflict check — bookkeeping, no server
# ---------------------------------------------------------------------------
class _FakeScroll:
def __init__(self, open_: bool = True) -> None:
self._server_cursor_open = open_
def _conflict_check(conn: informix_db.Connection, requester: object) -> None:
conn._check_scroll_cursor_conflict(requester)
def test_conflict_check_is_silent_with_no_scroll_cursor() -> None:
conn = informix_db.Connection.__new__(informix_db.Connection)
conn._open_scroll_cursor = None
_conflict_check(conn, object())
def test_conflict_check_forgets_a_collected_cursor() -> None:
"""The ref is weak so an abandoned scrollable cursor can still be
finalized. A dead ref means the slot is free."""
import weakref
conn = informix_db.Connection.__new__(informix_db.Connection)
victim = _FakeScroll()
conn._open_scroll_cursor = weakref.ref(victim)
del victim
_conflict_check(conn, object())
assert conn._open_scroll_cursor is None
def test_conflict_check_lets_the_owner_through() -> None:
"""Re-executing the *same* scrollable cursor is allowed — it closes
its own server-side cursor first."""
import weakref
conn = informix_db.Connection.__new__(informix_db.Connection)
owner = _FakeScroll()
conn._open_scroll_cursor = weakref.ref(owner)
_conflict_check(conn, owner)
def test_conflict_check_forgets_a_closed_cursor() -> None:
import weakref
conn = informix_db.Connection.__new__(informix_db.Connection)
done = _FakeScroll(open_=False)
conn._open_scroll_cursor = weakref.ref(done)
_conflict_check(conn, object())
assert conn._open_scroll_cursor is None
# ---------------------------------------------------------------------------
# Against a real server
# ---------------------------------------------------------------------------
@pytest.mark.integration
def test_second_statement_is_refused_while_scroll_cursor_is_open(
conn_params: ConnParams,
) -> None:
"""Used to be -285 for the new statement plus -267 for the scrollable
cursor. Now it's one error that names the cause, and the scrollable
cursor is untouched."""
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
first = scroll.fetch_first()
assert first is not None
other = conn.cursor()
with pytest.raises(informix_db.ProgrammingError, match="scrollable"):
other.execute("SELECT COUNT(*) FROM systables")
assert scroll.fetch_absolute(1) is not None, (
"the refused statement must not have disturbed the cursor"
)
scroll.close()
other.execute("SELECT COUNT(*) FROM systables")
assert other.fetchone() is not None, "slot must free on close"
@pytest.mark.integration
def test_executemany_is_refused_too(conn_params: ConnParams) -> None:
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_slot (k INT)")
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
scroll.fetch_first()
with pytest.raises(informix_db.ProgrammingError, match="scrollable"):
cur.executemany("INSERT INTO t_slot VALUES (?)", [(1,), (2,)])
scroll.close()
cur.executemany("INSERT INTO t_slot VALUES (?)", [(1,), (2,)])
cur.execute("SELECT COUNT(*) FROM t_slot")
assert cur.fetchone() == (2,)
@pytest.mark.integration
def test_scroll_cursor_can_be_re_executed(conn_params: ConnParams) -> None:
"""The owner is the one caller allowed past the conflict check, and
that is only safe because it closes its own server-side cursor
first. Without that it collides with itself."""
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
for _ in range(4):
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
assert scroll.fetch_first() is not None
scroll.close()
@pytest.mark.integration
def test_abandoning_a_scroll_cursor_frees_the_slot(
conn_params: ConnParams,
) -> None:
"""Dropping the last reference must let the connection be used again
the finalizer closes the cursor and the weak ref goes dead."""
import gc
with _connect(conn_params) as conn:
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
scroll.fetch_first()
del scroll
gc.collect()
cur = conn.cursor()
cur.execute("SELECT COUNT(*) FROM systables")
assert cur.fetchone() is not None
# ---------------------------------------------------------------------------
# Deferred cleanup
# ---------------------------------------------------------------------------
@pytest.mark.integration
def test_queued_cleanup_does_not_land_inside_a_running_statement(
conn_params: ConnParams,
) -> None:
"""Exactly what a cross-thread finalizer does: it lost the wire lock,
so it queued its CLOSE/RELEASE while another thread was mid-statement.
The flush used to happen before that thread's very next PDU — its own
CURNAME/NFETCH releasing the statement out from under it."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_defer (k INT)")
cur.executemany(
"INSERT INTO t_defer VALUES (?)", [(i,) for i in range(50)]
)
original = cur._read_describe_response
def enqueue_between_prepare_and_fetch() -> None:
result = original()
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
return result
cur._read_describe_response = enqueue_between_prepare_and_fetch
try:
cur.execute("SELECT k FROM t_defer ORDER BY k")
assert len(cur.fetchall()) == 50, (
"the queued cleanup released our own statement"
)
finally:
cur._read_describe_response = original
@pytest.mark.integration
def test_stale_queued_cleanup_does_not_kill_the_connection(
conn_params: ConnParams,
) -> None:
"""A queue entry goes stale whenever the cursor gets closed properly
between enqueue and flush. The server answers the leftover CLOSE with
-267, which is an OperationalError, which is in WIRE_ERRORS so a
stale entry used to force-close a healthy connection."""
with _connect(conn_params) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 1 tabid FROM systables")
cur.fetchall()
conn._enqueue_cleanup([_CLOSE_PDU, _RELEASE_PDU])
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None, "stale cleanup killed the connection"
assert not conn.closed
assert conn._pending_cleanup == [], "queue should have drained"
@pytest.mark.integration
def test_connection_closes_cleanly_with_a_scroll_cursor_open(
conn_params: ConnParams,
) -> None:
conn = _connect(conn_params)
scroll = conn.cursor(scrollable=True)
scroll.execute("SELECT tabid FROM systables ORDER BY tabid")
scroll.fetch_first()
with contextlib.suppress(Exception):
conn.close()
assert conn.closed
# ---------------------------------------------------------------------------
# The wire lock's blind spot
# ---------------------------------------------------------------------------
def test_wire_lock_reports_its_own_thread() -> None:
"""The whole point. ``RLock.acquire(blocking=False)`` returns True for
the owning thread, so the finalizer's probe could not tell "nobody is
using the wire" from "I am, right now, mid-statement"."""
from informix_db.connections import _WireLock
lock = _WireLock()
assert not lock.held_by_current_thread
with lock:
assert lock.held_by_current_thread
assert lock.acquire(blocking=False), "must still be reentrant"
lock.release()
assert lock.held_by_current_thread, "still held at depth 1"
assert not lock.held_by_current_thread
def test_wire_lock_is_not_held_by_other_threads() -> None:
import threading
from informix_db.connections import _WireLock
lock = _WireLock()
seen: list[bool] = []
entered = threading.Event()
done = threading.Event()
def holder() -> None:
with lock:
entered.set()
done.wait(5)
t = threading.Thread(target=holder)
t.start()
entered.wait(5)
seen.append(lock.held_by_current_thread)
done.set()
t.join(5)
assert seen == [False], "another thread's hold must not read as ours"
@pytest.mark.integration
def test_finalizer_defers_when_gc_runs_on_the_locking_thread(
conn_params: ConnParams,
) -> None:
"""GC fires on whatever thread allocated. When that is the thread
holding the wire lock, the finalizer must write nothing it used to
acquire the RLock reentrantly and send CLOSE/RELEASE into the running
statement, killing it with -208."""
import gc
with _connect(conn_params) as conn:
gc.disable()
try:
victim = conn.cursor(scrollable=True)
victim.execute("SELECT tabid FROM systables ORDER BY tabid")
victim.fetch_first()
# A reference cycle, so collection waits for gc rather than
# happening at the drop. Cycles are ordinary in Python.
cycle = [victim]
cycle.append(cycle)
del victim, cycle
writes: list[bytes] = []
original_write = conn._sock.write_all
conn._sock.write_all = lambda b: (
writes.append(b),
original_write(b),
)[1]
try:
with conn._wire_lock: # stand in for "mid-statement"
gc.collect()
assert writes == [], (
"finalizer wrote to the wire while a statement "
"owned it"
)
assert conn._pending_cleanup, (
"cleanup should have been deferred, not dropped"
)
finally:
conn._sock.write_all = original_write
finally:
gc.enable()
cur = conn.cursor()
cur.execute("SELECT COUNT(*) FROM systables")
assert cur.fetchone() is not None

333
tests/test_tls_traffic.py Normal file
View File

@ -0,0 +1,333 @@
"""End-to-end SQLI traffic over a real TLS socket.
``tests/test_tls.py`` covers the handshake. This covers what happens
*after* it: every byte of real SQLI traffic crossing genuine TLS records,
through the same codecs, reader, and cursor machinery the plain-socket
suite exercises.
That distinction matters because ``SSLSocket.recv`` is not
``socket.recv``. It returns at most one TLS record's worth of plaintext
however much you ask for, it can return fewer bytes than are available,
and plaintext buffered inside the SSL object is invisible to the OS. The
Phase 39 buffered reader asks for up to 64 KB per call and loops until
satisfied that loop is the thing which has to be right, and nothing in
the plain-socket suite puts the same pressure on it.
**Scope.** A TLS-terminating proxy in front of the plain SQLI listener
supplies the TLS half. This tests the driver's TLS path, which is the
half we own. It does *not* test IBM's server-side TLS listener: Informix
15 wants a PKCS#12 keystore whose stash the developer-edition image
rejects (``GSK_ERROR_BAD_KEYFILE_PASSWORD``), and that side is IBM's
code. Anything below ``ssl.wrap_socket`` is identical either way.
Skipped when ``openssl`` isn't on PATH — the proxy needs a certificate.
"""
from __future__ import annotations
import contextlib
import datetime
import decimal
import select
import shutil
import socket
import ssl
import subprocess
import tempfile
import threading
from pathlib import Path
import pytest
import informix_db
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
# ---------------------------------------------------------------------------
# TLS-terminating proxy
# ---------------------------------------------------------------------------
class _TlsProxy:
"""Accepts TLS, relays plaintext to the real Informix listener."""
def __init__(self, backend: tuple[str, int]) -> None:
self.backend = backend
self.tmpdir = tempfile.mkdtemp(prefix="ifx-tls-test-")
self.cert = str(Path(self.tmpdir) / "cert.pem")
key = str(Path(self.tmpdir) / "key.pem")
subprocess.run(
["openssl", "req", "-x509", "-newkey", "rsa:2048", "-nodes",
"-keyout", key, "-out", self.cert, "-days", "1",
"-subj", "/CN=127.0.0.1"],
check=True, capture_output=True,
)
self._ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
self._ctx.load_cert_chain(self.cert, key)
self._sock = socket.socket()
self._sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._sock.bind(("127.0.0.1", 0))
self._sock.listen(64)
self.port: int = self._sock.getsockname()[1]
self._stop = threading.Event()
threading.Thread(target=self._serve, daemon=True).start()
def _serve(self) -> None:
self._sock.settimeout(0.5)
while not self._stop.is_set():
try:
raw, _ = self._sock.accept()
except (TimeoutError, OSError):
continue
threading.Thread(
target=self._handle, args=(raw,), daemon=True
).start()
def _handle(self, raw: socket.socket) -> None:
try:
client = self._ctx.wrap_socket(raw, server_side=True)
except (ssl.SSLError, OSError):
with contextlib.suppress(OSError):
raw.close()
return
try:
upstream = socket.create_connection(self.backend, timeout=20)
except OSError:
with contextlib.suppress(OSError):
client.close()
return
try:
self._pump(client, upstream)
finally:
for s in (client, upstream):
with contextlib.suppress(OSError):
s.close()
@staticmethod
def _pump(a: socket.socket, b: socket.socket) -> None:
# Drain the SSL object's own buffer before consulting select():
# select only sees the OS socket, so already-decrypted bytes
# sitting inside the SSL object would stall the relay.
socks = [a, b]
while True:
pending = [s for s in socks
if isinstance(s, ssl.SSLSocket) and s.pending()]
ready = pending or select.select(socks, [], [], 1.0)[0]
for s in ready:
other = b if s is a else a
try:
data = s.recv(65536)
except (ssl.SSLError, OSError):
return
if not data:
return
try:
other.sendall(data)
except OSError:
return
def close(self) -> None:
self._stop.set()
with contextlib.suppress(OSError):
self._sock.close()
shutil.rmtree(self.tmpdir, ignore_errors=True)
@pytest.fixture(scope="module")
def tls_proxy(conn_params: ConnParams):
if shutil.which("openssl") is None:
pytest.skip("openssl not on PATH; needed to generate a test cert")
proxy = _TlsProxy((conn_params.host, conn_params.port))
try:
yield proxy
finally:
proxy.close()
def _client_ctx(proxy: _TlsProxy) -> ssl.SSLContext:
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.load_verify_locations(proxy.cert)
ctx.check_hostname = False # self-signed CN=127.0.0.1
return ctx
def _connect(proxy: _TlsProxy, conn_params: ConnParams, **kw):
return informix_db.connect(
host="127.0.0.1",
port=proxy.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=20.0,
read_timeout=45.0,
tls=_client_ctx(proxy),
**kw,
)
# ---------------------------------------------------------------------------
# Traffic
# ---------------------------------------------------------------------------
def test_query_over_tls(tls_proxy, conn_params: ConnParams) -> None:
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("SELECT FIRST 3 tabname FROM systables ORDER BY tabid")
assert len(cur.fetchall()) == 3
assert conn.server_version, "server_version empty over TLS"
def test_type_round_trip_over_tls(tls_proxy, conn_params: ConnParams) -> None:
"""The types that gave us framing bugs, every byte through TLS."""
ts = datetime.datetime(2026, 8, 31, 12, 30, 15, 120000)
row = (
2001, "PackageRoot", None, "/content/package", 77,
decimal.Decimal("1234567890123456"), True, ts, "nch",
)
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute(
"CREATE TEMP TABLE t_tls_types ("
" a INT8 NOT NULL, k LVARCHAR(512), d LVARCHAR(512),"
" v LVARCHAR(1024), n INT8, dec16 DECIMAL(16), b BOOLEAN,"
" t DATETIME YEAR TO FRACTION(5), c NCHAR(6))"
)
cur.execute(
"INSERT INTO t_tls_types VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", row
)
cur.execute("SELECT a, k, d, v, n, dec16, b, t, c FROM t_tls_types")
assert cur.fetchone() == row
@pytest.mark.parametrize("size", [1, 4096, 16383, 16384, 16385, 32000])
def test_payload_spans_tls_record_boundary(
tls_proxy, conn_params: ConnParams, size: int
) -> None:
"""A TLS record holds ~16 KB, so these straddle the boundary where a
single ``recv`` stops being enough."""
payload = "x" * size
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_tls_big (k INT, v LVARCHAR(32000))")
cur.execute("INSERT INTO t_tls_big VALUES (?, ?)", (1, payload))
cur.execute("SELECT v, k FROM t_tls_big")
assert cur.fetchone() == (payload, 1)
@pytest.mark.parametrize("n", [500, 5000])
def test_bulk_fetch_over_tls(
tls_proxy, conn_params: ConnParams, n: int
) -> None:
"""Total bytes far beyond both one TLS record and the reader's 64 KB
recv budget, so the top-up loop runs many times."""
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_tls_bulk (k INT, v VARCHAR(240))")
cur.executemany(
"INSERT INTO t_tls_bulk VALUES (?, ?)",
[(i, f"row{i}-" + "y" * 200) for i in range(n)],
)
cur.execute("SELECT k, v FROM t_tls_bulk ORDER BY k")
rows = cur.fetchall()
assert len(rows) == n
assert rows[0][0] == 0
assert rows[-1][0] == n - 1
def test_error_recovery_over_tls(tls_proxy, conn_params: ConnParams) -> None:
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
cur.execute("CREATE TEMP TABLE t_tls_dup (k INT PRIMARY KEY)")
cur.execute("INSERT INTO t_tls_dup VALUES (1)")
for _ in range(4):
with pytest.raises(informix_db.Error):
cur.execute("INSERT INTO t_tls_dup VALUES (1)")
with pytest.raises(informix_db.Error):
cur.execute("SELECT * FROM t_tls_no_such_table_xyz")
cur.execute("SELECT k FROM t_tls_dup")
assert cur.fetchall() == [(1,)]
def test_concurrent_tls_connections(tls_proxy, conn_params: ConnParams) -> None:
"""Separate TLS sessions must not cross data."""
failures: list[str] = []
lock = threading.Lock()
def worker(tid: int) -> None:
tag = f"t{tid}-{'z' * 12}"
try:
with _connect(tls_proxy, conn_params, autocommit=True) as conn:
cur = conn.cursor()
for r in range(6):
token = tid * 1000 + r
cur.execute(
"SELECT FIRST 1 ?::INT, ?::VARCHAR(24) FROM systables",
(token, tag),
)
if cur.fetchone() != (token, tag):
with lock:
failures.append(f"thread {tid}: crossed data")
return
except Exception as exc:
with lock:
failures.append(f"thread {tid}: {type(exc).__name__}: {exc}")
threads = [threading.Thread(target=worker, args=(i,)) for i in range(4)]
for t in threads:
t.start()
for t in threads:
t.join()
assert not failures, failures
# ---------------------------------------------------------------------------
# Negative cases — misuse must fail cleanly, never hang or downgrade
# ---------------------------------------------------------------------------
def test_tls_client_against_plaintext_port_fails(
tls_proxy, conn_params: ConnParams
) -> None:
with pytest.raises(informix_db.Error):
informix_db.connect(
host=conn_params.host, port=conn_params.port,
user=conn_params.user, password=conn_params.password,
database=conn_params.database, server=conn_params.server,
connect_timeout=10.0, read_timeout=10.0,
tls=_client_ctx(tls_proxy),
)
def test_plaintext_client_against_tls_port_fails(
tls_proxy, conn_params: ConnParams
) -> None:
"""Must raise rather than hang — a stalled handshake is the failure
mode that looks like a dead application."""
with pytest.raises(informix_db.Error):
informix_db.connect(
host="127.0.0.1", port=tls_proxy.port,
user=conn_params.user, password=conn_params.password,
database=conn_params.database, server=conn_params.server,
connect_timeout=10.0, read_timeout=10.0,
)
def test_verification_rejects_self_signed(
tls_proxy, conn_params: ConnParams
) -> None:
"""`tls=True` disables verification by design; a caller-supplied
verifying context must still reject an untrusted cert."""
strict = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
strict.check_hostname = True
strict.verify_mode = ssl.CERT_REQUIRED
with pytest.raises(informix_db.Error):
informix_db.connect(
host="127.0.0.1", port=tls_proxy.port,
user=conn_params.user, password=conn_params.password,
database=conn_params.database, server=conn_params.server,
connect_timeout=10.0, read_timeout=10.0, tls=strict,
)

View File

@ -0,0 +1,191 @@
"""Transaction control run as SQL, and the flag that didn't notice.
``Connection._in_transaction`` decides whether ``commit()`` and
``rollback()`` send anything at all, and the pool reads it to decide
whether a returned connection needs cleaning up. It was maintained
solely by the driver's own implicit ``SQ_BEGIN``, so a caller who wrote
``cursor.execute("BEGIN WORK")`` an entirely reasonable thing to
write walked straight past it.
With autocommit on, nothing stopped that statement reaching the server.
A transaction opened, the flag stayed False, and ``rollback()`` returned
successfully having sent nothing. The rows it was asked to discard were
still there. The connection then went back to the pool holding an open
transaction and its locks, because the pool's cleanup is guarded by the
same flag.
With autocommit off it failed instead, and for a sillier reason: the
driver's implicit ``SQ_BEGIN`` fired first, so the caller's ``BEGIN
WORK`` got ``-535``, "already in transaction". The driver and the user
competing to open the same transaction, and the user losing.
The server labels these statements: type 34 for BEGIN, 35 for COMMIT, 36
for ROLLBACK, with and without the ``WORK`` keyword. JDBC reads the same
three values off the describe and calls ``setTxBeginState`` /
``setTxEndState``. It also calls ``initiateTransaction`` *after* the
describe rather than before, which is what makes the skip possible
until the describe lands you can't know the statement is transaction
control.
"""
from __future__ import annotations
import contextlib
import pytest
import informix_db
from informix_db.cursors import _TX_CONTROL_TYPES
from tests.conftest import ConnParams
pytestmark = pytest.mark.integration
def _connect(conn_params: ConnParams, **kw) -> informix_db.Connection:
return informix_db.connect(
host=conn_params.host,
port=conn_params.port,
user=conn_params.user,
password=conn_params.password,
database=conn_params.database,
server=conn_params.server,
connect_timeout=10.0,
read_timeout=25.0,
**kw,
)
def test_transaction_control_types_are_what_the_server_says(
logged_db_params: ConnParams,
) -> None:
"""Pin the three constants against a live server rather than trusting
the decompiled source. Both spellings must map to the same type."""
with _connect(logged_db_params, autocommit=True) as conn:
cur = conn.cursor()
seen = {}
for sql in ("BEGIN WORK", "COMMIT WORK", "ROLLBACK WORK",
"BEGIN", "COMMIT", "ROLLBACK"):
with conn._wire_lock:
conn._send_pdu(
cur._build_prepare_pdu(sql, num_qmarks=0),
statement_boundary=True,
)
cur._read_describe_response()
cur._release_after_failure()
seen[sql] = cur._statement_type
assert seen["BEGIN WORK"] == seen["BEGIN"] == 34
assert seen["COMMIT WORK"] == seen["COMMIT"] == 35
assert seen["ROLLBACK WORK"] == seen["ROLLBACK"] == 36
assert set(seen.values()) == _TX_CONTROL_TYPES
def test_rollback_after_sql_begin_actually_rolls_back(
logged_db_params: ConnParams,
) -> None:
"""The data-loss case. rollback() reported success and sent nothing,
so the row it was asked to discard survived."""
with _connect(logged_db_params, autocommit=True) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate")
cur.execute("CREATE TABLE t_txstate (k INT)")
try:
cur.execute("BEGIN WORK")
assert conn._in_transaction, "SQL BEGIN must set the flag"
cur.execute("INSERT INTO t_txstate VALUES (1)")
conn.rollback()
cur.execute("SELECT COUNT(*) FROM t_txstate")
assert cur.fetchone() == (0,), "rollback() was a silent no-op"
assert not conn._in_transaction
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate")
def test_sql_commit_clears_the_flag(logged_db_params: ConnParams) -> None:
"""The mirror image: with the flag stuck True after a SQL COMMIT, the
next rollback() would send SQ_RBWORK with no transaction open and
draw -255."""
with _connect(logged_db_params, autocommit=True) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate2")
cur.execute("CREATE TABLE t_txstate2 (k INT)")
try:
cur.execute("BEGIN WORK")
cur.execute("INSERT INTO t_txstate2 VALUES (1)")
cur.execute("COMMIT WORK")
assert not conn._in_transaction, "SQL COMMIT must clear the flag"
conn.rollback() # must be a no-op, not a -255
cur.execute("SELECT COUNT(*) FROM t_txstate2")
assert cur.fetchone() == (1,), "the committed row must survive"
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate2")
def test_sql_begin_does_not_collide_with_the_implicit_one(
logged_db_params: ConnParams,
) -> None:
"""Non-autocommit. The driver's implicit SQ_BEGIN used to fire first
and the caller's BEGIN WORK then got -535."""
with _connect(logged_db_params, autocommit=False) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate3")
conn.commit()
cur.execute("CREATE TABLE t_txstate3 (k INT)")
conn.commit()
try:
cur.execute("BEGIN WORK")
cur.execute("INSERT INTO t_txstate3 VALUES (1)")
conn.rollback()
cur.execute("SELECT COUNT(*) FROM t_txstate3")
assert cur.fetchone() == (0,)
conn.commit()
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate3")
conn.commit()
def test_ordinary_dml_still_opens_a_transaction(
logged_db_params: ConnParams,
) -> None:
"""_ensure_transaction moved from before the PREPARE to after the
describe. It still has to fire for everything that isn't transaction
control, or non-autocommit DML runs outside a transaction."""
with _connect(logged_db_params, autocommit=False) as conn:
cur = conn.cursor()
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate4")
conn.commit()
cur.execute("CREATE TABLE t_txstate4 (k INT)")
conn.commit()
try:
assert not conn._in_transaction
cur.execute("INSERT INTO t_txstate4 VALUES (1)")
assert conn._in_transaction, "DML must open a transaction"
conn.rollback()
cur.execute("SELECT COUNT(*) FROM t_txstate4")
assert cur.fetchone() == (0,)
conn.commit()
finally:
with contextlib.suppress(Exception):
cur.execute("DROP TABLE t_txstate4")
conn.commit()
def test_a_failed_transaction_statement_does_not_move_the_flag(
logged_db_params: ConnParams,
) -> None:
"""The sync runs on the success path only. A COMMIT that the server
rejects has not ended anything."""
with _connect(logged_db_params, autocommit=True) as conn:
cur = conn.cursor()
assert not conn._in_transaction
with pytest.raises(informix_db.Error):
cur.execute("COMMIT WORK") # -255, nothing to commit
assert not conn._in_transaction
cur.execute("SELECT FIRST 1 tabid FROM systables")
assert cur.fetchone() is not None

2
uv.lock generated
View File

@ -34,7 +34,7 @@ wheels = [
[[package]]
name = "informix-driver"
version = "2026.8.31"
version = "2026.9.3"
source = { editable = "." }
[package.optional-dependencies]