Ryan Malloy
9616ddbc0a
More data corruption from the same field report as 2026.05.08.2. Anyone
with LVARCHAR columns should upgrade: 2026.08.27 is not safe for them.
Two independent errors in the LVARCHAR envelope, both shifting every
column selected after it.
1. A phantom pad byte. We appended an even-byte pad when the value
length was odd. There is no pad. Wire capture (12.10), INT8 /
LVARCHAR / INT8:
00 01 00 00 07 d1 00 00 00 00 a = INT8 2001
00 null indicator
00 00 00 0b length 11
50 61 63 6b 61 67 65 52 6f 6f 74 "PackageRoot" (odd)
00 01 00 00 00 0a 00 00 00 00 b = INT8 10, starts immediately
2. A missing length on NULL. We returned as soon as the indicator said
NULL, leaving its 4-byte length unread. The length belongs to the
envelope and is always present:
'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes
NULL -> 01 | 00 00 00 00 5 bytes
'' -> 00 | 00 00 00 00 5 bytes
NULL and empty string differ only in the indicator byte.
Reported symptom: INT8 10 decoding as 2560 (the same value shifted one
byte left), strings losing their first character, and IndexError or
"INT8 payload too short" once the drift ran off the payload. The
reporter isolated it by reordering columns in the projection — right
when first, wrong when later. That's the fingerprint of positional
drift and a genuinely good diagnostic.
Missed by 247 tests because the only LVARCHAR fixture was 'lv value':
8 characters, even, never NULL. Neither faulty branch ever ran. Same
gap shape as last time — code paths covered, the data reaching them
not. New tests vary parity (0,1,2,3,11,255,256), cover NULL and empty
separately, always place a column AFTER the LVARCHAR, and rotate the
projection through every position.
Separately, DATETIME lost sub-second precision on INSERT. The encoder
emitted YEAR TO SECOND unconditionally, so binding a datetime with
microseconds into a FRACTION(n) column stored zeros, silently. Reads
were always right, so it only went missing on the way in. Now widens
to FRACTION(5) when microsecond is non-zero and keeps the original
encoding otherwise, so the well-exercised path stays byte-identical.
Binding into a narrower column still truncates server-side.
Also: server_version reported 9.56 for a 12.10 server, which reads like
a client-SDK version. The login response only carries the internal
protocol version, and documenting that didn't make the name less
misleading. server_version now returns the release via one cached
DBINFO query; server_version_internal returns the raw login string.
281/281 integration on 15, 14.10 and 12.10; 123 unit tests.
250 lines
8.7 KiB
Python
250 lines
8.7 KiB
Python
"""Regression tests for LVARCHAR tuple framing, reported 2026-08-31.
|
|
|
|
Two independent framing errors, both of which shifted every column that
|
|
followed an LVARCHAR:
|
|
|
|
1. **A phantom pad byte.** We appended an even-byte pad when the value
|
|
length was odd. There is no pad — the next column begins immediately
|
|
after the last content byte.
|
|
2. **A missing length on NULL.** We returned as soon as the null
|
|
indicator said NULL, leaving the 4-byte length field unread. The
|
|
length is part of the envelope and is always present.
|
|
|
|
Both survived a 247-test suite because the only LVARCHAR fixture used
|
|
``'lv value'`` — 8 characters, even, and never NULL. Neither faulty
|
|
branch ever executed.
|
|
|
|
The tests below therefore vary length parity and nullness deliberately,
|
|
and always place a column *after* the LVARCHAR, because the damage lands
|
|
downstream: a trailing LVARCHAR can be mis-sized with no visible effect.
|
|
|
|
Wire evidence (Informix 12.10) for INT8 / LVARCHAR / INT8:
|
|
|
|
'odd' -> 00 | 00 00 00 03 | 6f 64 64 8 bytes
|
|
NULL -> 01 | 00 00 00 00 5 bytes
|
|
'' -> 00 | 00 00 00 00 5 bytes
|
|
|
|
NULL and empty string differ only in the indicator byte.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import datetime
|
|
|
|
import pytest
|
|
|
|
import informix_db
|
|
from tests.conftest import ConnParams
|
|
|
|
pytestmark = pytest.mark.integration
|
|
|
|
|
|
def _connect(conn_params: ConnParams) -> informix_db.Connection:
|
|
return informix_db.connect(
|
|
host=conn_params.host,
|
|
port=conn_params.port,
|
|
user=conn_params.user,
|
|
password=conn_params.password,
|
|
database=conn_params.database,
|
|
server=conn_params.server,
|
|
connect_timeout=10.0,
|
|
read_timeout=10.0,
|
|
)
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Length parity — the phantom pad byte
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
@pytest.mark.parametrize(
|
|
"text",
|
|
[
|
|
"", # 0 — even, empty
|
|
"a", # 1 — ODD
|
|
"ab", # 2
|
|
"abc", # 3 — ODD
|
|
"PackageRoot", # 11 — ODD, the reported value
|
|
"lv value", # 8 — the old fixture that hid the bug
|
|
"x" * 255, # 255 — ODD, spans a length byte boundary
|
|
"y" * 256, # 256
|
|
],
|
|
)
|
|
def test_lvarchar_length_parity_does_not_shift_next_column(
|
|
conn_params: ConnParams, text: str
|
|
) -> None:
|
|
"""A trailing sentinel column catches any over- or under-read."""
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
cur.execute(
|
|
"CREATE TEMP TABLE t_lv_parity "
|
|
"(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL, c VARCHAR(8))"
|
|
)
|
|
cur.execute(
|
|
"INSERT INTO t_lv_parity VALUES (?, ?, ?, ?)",
|
|
(2001, text, 10, "tail"),
|
|
)
|
|
cur.execute("SELECT a, s, b, c FROM t_lv_parity")
|
|
assert cur.fetchone() == (2001, text, 10, "tail")
|
|
|
|
|
|
def test_odd_length_lvarchar_reproduces_the_report(
|
|
conn_params: ConnParams,
|
|
) -> None:
|
|
"""The exact failure shape: INT8 decoded as its own value shifted one
|
|
byte left (10 -> 2560), and the following string losing a character."""
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
cur.execute(
|
|
"CREATE TEMP TABLE t_lv_report "
|
|
"(a INT8 NOT NULL, k LVARCHAR(512) NOT NULL,"
|
|
" b INT8 NOT NULL, v LVARCHAR(1024))"
|
|
)
|
|
cur.execute(
|
|
"INSERT INTO t_lv_report VALUES (?, ?, ?, ?)",
|
|
(2001, "PackageRoot", 10, "/content/package"),
|
|
)
|
|
cur.execute("SELECT a, k, b, v FROM t_lv_report")
|
|
row = cur.fetchone()
|
|
assert row == (2001, "PackageRoot", 10, "/content/package")
|
|
assert row[2] != 2560, "INT8 shifted one byte left"
|
|
assert row[3].startswith("/"), "leading character lost"
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# NULL — the missing length field
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
def test_null_lvarchar_does_not_shift_next_column(
|
|
conn_params: ConnParams,
|
|
) -> None:
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
cur.execute(
|
|
"CREATE TEMP TABLE t_lv_null "
|
|
"(a INT8 NOT NULL, s LVARCHAR(512), b INT8 NOT NULL)"
|
|
)
|
|
cur.execute("INSERT INTO t_lv_null VALUES (?, ?, ?)", (3001, None, 20))
|
|
cur.execute("SELECT a, s, b FROM t_lv_null")
|
|
assert cur.fetchone() == (3001, None, 20)
|
|
|
|
|
|
def test_null_and_empty_lvarchar_are_distinguished(
|
|
conn_params: ConnParams,
|
|
) -> None:
|
|
"""They differ only in the indicator byte, so it's easy to conflate."""
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
cur.execute(
|
|
"CREATE TEMP TABLE t_lv_ne (k INT, s LVARCHAR(64), tail INT)"
|
|
)
|
|
cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (1, None, 111))
|
|
cur.execute("INSERT INTO t_lv_ne VALUES (?, ?, ?)", (2, "", 222))
|
|
cur.execute("SELECT k, s, tail FROM t_lv_ne ORDER BY k")
|
|
assert cur.fetchall() == [(1, None, 111), (2, "", 222)]
|
|
|
|
|
|
def test_consecutive_null_lvarchars(conn_params: ConnParams) -> None:
|
|
"""Each NULL under-read by 4 bytes, so several in a row compound."""
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
cur.execute(
|
|
"CREATE TEMP TABLE t_lv_many_null "
|
|
"(a INT8 NOT NULL, s1 LVARCHAR(256), s2 LVARCHAR(256),"
|
|
" s3 LVARCHAR(256), b INT8 NOT NULL)"
|
|
)
|
|
cur.execute(
|
|
"INSERT INTO t_lv_many_null VALUES (?, ?, ?, ?, ?)",
|
|
(4001, None, None, None, 40),
|
|
)
|
|
cur.execute("SELECT a, s1, s2, s3, b FROM t_lv_many_null")
|
|
assert cur.fetchone() == (4001, None, None, None, 40)
|
|
|
|
|
|
# --------------------------------------------------------------------------
|
|
# Wide, mixed shape — several LVARCHARs interleaved with other types
|
|
# --------------------------------------------------------------------------
|
|
|
|
|
|
_WIDE_COLS = [
|
|
"tag", "key_txt", "def_txt", "cur_txt", "note_txt",
|
|
"ver", "made_by", "ident", "made_on",
|
|
]
|
|
_WIDE_ROW = (
|
|
"Gadget",
|
|
"PackageRoot", # 11 — ODD
|
|
None, # NULL
|
|
"/content/package", # 16
|
|
"z", # 1 — ODD
|
|
77,
|
|
"maker",
|
|
3001,
|
|
datetime.datetime(2026, 8, 31, 12, 30, 15, 120000),
|
|
)
|
|
|
|
|
|
def _make_wide(cur) -> None:
|
|
cur.execute(
|
|
"CREATE TEMP TABLE t_lv_wide ("
|
|
" tag VARCHAR(32) NOT NULL,"
|
|
" key_txt LVARCHAR(512) NOT NULL,"
|
|
" def_txt LVARCHAR(1024),"
|
|
" cur_txt LVARCHAR(1024),"
|
|
" note_txt LVARCHAR(1024),"
|
|
" ver INT8 DEFAULT 0 NOT NULL,"
|
|
" made_by VARCHAR(100),"
|
|
" ident INT8 NOT NULL,"
|
|
" made_on DATETIME YEAR TO FRACTION(5))"
|
|
)
|
|
cur.execute(
|
|
f"INSERT INTO t_lv_wide VALUES ({', '.join(['?'] * len(_WIDE_COLS))})",
|
|
_WIDE_ROW,
|
|
)
|
|
|
|
|
|
def test_wide_mixed_row(conn_params: ConnParams) -> None:
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
_make_wide(cur)
|
|
cur.execute(f"SELECT {', '.join(_WIDE_COLS)} FROM t_lv_wide")
|
|
assert cur.fetchone() == _WIDE_ROW
|
|
|
|
|
|
@pytest.mark.parametrize("shift", range(len(_WIDE_COLS)))
|
|
def test_wide_row_survives_column_reordering(
|
|
conn_params: ConnParams, shift: int
|
|
) -> None:
|
|
"""The reporter isolated this by reordering columns — values were
|
|
correct first in the list and wrong later on. Rotating the projection
|
|
exercises every position for every type."""
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
_make_wide(cur)
|
|
order = _WIDE_COLS[shift:] + _WIDE_COLS[:shift]
|
|
cur.execute(f"SELECT {', '.join(order)} FROM t_lv_wide")
|
|
expected = tuple(_WIDE_ROW[_WIDE_COLS.index(c)] for c in order)
|
|
assert cur.fetchone() == expected
|
|
|
|
|
|
def test_select_star_wide_row(conn_params: ConnParams) -> None:
|
|
"""``SELECT *`` raised IndexError once enough LVARCHARs accumulated."""
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
_make_wide(cur)
|
|
cur.execute("SELECT * FROM t_lv_wide")
|
|
row = cur.fetchone()
|
|
names = [d[0] for d in cur.description]
|
|
assert row == tuple(_WIDE_ROW[_WIDE_COLS.index(n)] for n in names)
|
|
|
|
|
|
def test_repeated_lvarchar_columns(conn_params: ConnParams) -> None:
|
|
"""Selecting the same LVARCHAR twice doubles any per-column drift."""
|
|
with _connect(conn_params) as conn:
|
|
cur = conn.cursor()
|
|
_make_wide(cur)
|
|
cur.execute(
|
|
"SELECT ident, key_txt, ident, key_txt, ver FROM t_lv_wide"
|
|
)
|
|
assert cur.fetchone() == (3001, "PackageRoot", 3001, "PackageRoot", 77)
|