Skip to content

bpo-44822: Don't truncate strs with embedded NULL chars returned by sqlite3 UDF callbacks #27588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Aug 5, 2021
Merged

bpo-44822: Don't truncate strs with embedded NULL chars returned by sqlite3 UDF callbacks #27588

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2241795
bpo-44822: Pass string size to sqlite3_result_text()
Aug 3, 2021
58d3244
Raise OverflowError if size > INT_MAX
Aug 4, 2021
73639f1
Add test
Aug 4, 2021
60d94f9
Add NEWS entry
Aug 4, 2021
10fbc4b
Don't capitalize error string
Aug 4, 2021
a483052
Add aggregate test
Aug 4, 2021
55963ca
Address review: reword NEWS entry
Aug 4, 2021
7a2c279
Apply suggestions from code review
serhiy-storchaka Aug 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions Lib/sqlite3/test/userfunctions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,8 @@ def wrapper(self, *args, **kwargs):

def func_returntext():
return "foo"
def func_returntextwithnull():
return "1\x002"
def func_returnunicode():
return "bar"
def func_returnint():
Expand Down Expand Up @@ -163,11 +165,21 @@ def step(self, val):
def finalize(self):
return self.val

class AggrText:
def __init__(self):
self.txt = ""
def step(self, txt):
self.txt = self.txt + txt
def finalize(self):
return self.txt


class FunctionTests(unittest.TestCase):
def setUp(self):
self.con = sqlite.connect(":memory:")

self.con.create_function("returntext", 0, func_returntext)
self.con.create_function("returntextwithnull", 0, func_returntextwithnull)
self.con.create_function("returnunicode", 0, func_returnunicode)
self.con.create_function("returnint", 0, func_returnint)
self.con.create_function("returnfloat", 0, func_returnfloat)
Expand Down Expand Up @@ -211,6 +223,12 @@ def test_func_return_text(self):
self.assertEqual(type(val), str)
self.assertEqual(val, "foo")

def test_func_return_text_with_null_char(self):
cur = self.con.cursor()
res = cur.execute("select returntextwithnull()").fetchone()[0]
self.assertEqual(type(res), str)
self.assertEqual(res, "1\x002")

def test_func_return_unicode(self):
cur = self.con.cursor()
cur.execute("select returnunicode()")
Expand Down Expand Up @@ -390,6 +408,7 @@ def setUp(self):
self.con.create_aggregate("checkType", 2, AggrCheckType)
self.con.create_aggregate("checkTypes", -1, AggrCheckTypes)
self.con.create_aggregate("mysum", 1, AggrSum)
self.con.create_aggregate("aggtxt", 1, AggrText)

def tearDown(self):
#self.cur.close()
Expand Down Expand Up @@ -486,6 +505,15 @@ def test_aggr_no_match(self):
val = cur.fetchone()[0]
self.assertIsNone(val)

def test_aggr_text(self):
cur = self.con.cursor()
for txt in ["foo", "1\x002"]:
with self.subTest(txt=txt):
cur.execute("select aggtxt(?) from test", (txt,))
val = cur.fetchone()[0]
self.assertEqual(val, txt)


class AuthorizerTests(unittest.TestCase):
@staticmethod
def authorizer_cb(action, arg1, arg2, dbname, source):
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/Library/2021-08-04-12-29-00.bpo-44822.zePNXA.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
:mod:`sqlite3` now raises :exc:`OverflowError` if an user-defined function
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually sqlite3 overwrites OverflowError with other exception. But it does not matter, the most important part of this PR is that result strings with NUL are now correctly supported.

Something like (please correct my English):

Suggested change
:mod:`sqlite3` now raises :exc:`OverflowError` if an user-defined function
Fix support of user functions and aggregates returning string containing NUL in :mod:`sqlite3`.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually sqlite3 overwrites OverflowError with other exception.

Ah, yes, there should be some exception chaining in the UDF callbacks. I'll open an issue for it.

Anyway, how does the reworded NEWS entry look?

returns a :class:`str` larger than ``INT_MAX`` bytes. Patch by Erlend E.
Aasland.
13 changes: 10 additions & 3 deletions Modules/_sqlite/connection.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -519,10 +519,17 @@ _pysqlite_set_result(sqlite3_context* context, PyObject* py_val)
} else if (PyFloat_Check(py_val)) {
sqlite3_result_double(context, PyFloat_AsDouble(py_val));
} else if (PyUnicode_Check(py_val)) {
const char *str = PyUnicode_AsUTF8(py_val);
if (str == NULL)
Py_ssize_t sz;
const char *str = PyUnicode_AsUTF8AndSize(py_val, &sz);
if (str == NULL) {
return -1;
sqlite3_result_text(context, str, -1, SQLITE_TRANSIENT);
}
if (sz > INT_MAX) {
PyErr_SetString(PyExc_OverflowError,
"string is longer than INT_MAX bytes");
return -1;
}
sqlite3_result_text(context, str, (int)sz, SQLITE_TRANSIENT);
} else if (PyObject_CheckBuffer(py_val)) {
Py_buffer view;
if (PyObject_GetBuffer(py_val, &view, PyBUF_SIMPLE) != 0) {
Expand Down