Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Commit ee47543

Browse filesBrowse files
Erlend Egeberg Aaslandpalavivkumaraditya303JelleZijlstra
authored
gh-69093: Support basic incremental I/O to blobs in sqlite3 (GH-30680)
Authored-by: Aviv Palivoda <palaviv@gmail.com> Co-authored-by: Erlend E. Aasland <erlend.aasland@innova.no> Co-authored-by: palaviv <palaviv@gmail.com> Co-authored-by: Kumar Aditya <59607654+kumaraditya303@users.noreply.github.com> Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
1 parent c9d41bc commit ee47543
Copy full SHA for ee47543

File tree

16 files changed

+989
-7
lines changed
Filter options

16 files changed

+989
-7
lines changed

‎Doc/includes/sqlite3/blob.py

Copy file name to clipboard
+12Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import sqlite3
2+
3+
con = sqlite3.connect(":memory:")
4+
con.execute("create table test(blob_col blob)")
5+
con.execute("insert into test(blob_col) values (zeroblob(10))")
6+
7+
blob = con.blobopen("test", "blob_col", 1)
8+
blob.write(b"Hello")
9+
blob.write(b"World")
10+
blob.seek(0)
11+
print(blob.read()) # will print b"HelloWorld"
12+
blob.close()

‎Doc/library/sqlite3.rst

Copy file name to clipboardExpand all lines: Doc/library/sqlite3.rst
+66Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -394,6 +394,20 @@ Connection Objects
394394
supplied, this must be a callable returning an instance of :class:`Cursor`
395395
or its subclasses.
396396

397+
.. method:: blobopen(table, column, row, /, *, readonly=False, name="main")
398+
399+
Open a :class:`Blob` handle to the :abbr:`BLOB (Binary Large OBject)`
400+
located in row *row*, column *column*, table *table* of database *name*.
401+
When *readonly* is :const:`True` the blob is opened without write
402+
permissions.
403+
404+
.. note::
405+
406+
The blob size cannot be changed using the :class:`Blob` class.
407+
Use the SQL function ``zeroblob`` to create a blob with a fixed size.
408+
409+
.. versionadded:: 3.11
410+
397411
.. method:: commit()
398412

399413
This method commits the current transaction. If you don't call this method,
@@ -1088,6 +1102,58 @@ Exceptions
10881102
transactions turned off. It is a subclass of :exc:`DatabaseError`.
10891103

10901104

1105+
.. _sqlite3-blob-objects:
1106+
1107+
Blob Objects
1108+
------------
1109+
1110+
.. versionadded:: 3.11
1111+
1112+
.. class:: Blob
1113+
1114+
A :class:`Blob` instance is a :term:`file-like object` that can read and write
1115+
data in an SQLite :abbr:`BLOB (Binary Large OBject)`. Call ``len(blob)`` to
1116+
get the size (number of bytes) of the blob.
1117+
1118+
.. method:: close()
1119+
1120+
Close the blob.
1121+
1122+
The blob will be unusable from this point onward. An
1123+
:class:`~sqlite3.Error` (or subclass) exception will be raised if any
1124+
further operation is attempted with the blob.
1125+
1126+
.. method:: read(length=-1, /)
1127+
1128+
Read *length* bytes of data from the blob at the current offset position.
1129+
If the end of the blob is reached, the data up to
1130+
:abbr:`EOF (End of File)` will be returned. When *length* is not
1131+
specified, or is negative, :meth:`~Blob.read` will read until the end of
1132+
the blob.
1133+
1134+
.. method:: write(data, /)
1135+
1136+
Write *data* to the blob at the current offset. This function cannot
1137+
change the blob length. Writing beyond the end of the blob will raise
1138+
:exc:`ValueError`.
1139+
1140+
.. method:: tell()
1141+
1142+
Return the current access position of the blob.
1143+
1144+
.. method:: seek(offset, origin=os.SEEK_SET, /)
1145+
1146+
Set the current access position of the blob to *offset*. The *origin*
1147+
argument defaults to :data:`os.SEEK_SET` (absolute blob positioning).
1148+
Other values for *origin* are :data:`os.SEEK_CUR` (seek relative to the
1149+
current position) and :data:`os.SEEK_END` (seek relative to the blob’s
1150+
end).
1151+
1152+
:class:`Blob` example:
1153+
1154+
.. literalinclude:: ../includes/sqlite3/blob.py
1155+
1156+
10911157
.. _sqlite3-types:
10921158

10931159
SQLite and Python types

‎Doc/whatsnew/3.11.rst

Copy file name to clipboardExpand all lines: Doc/whatsnew/3.11.rst
+4Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -393,6 +393,10 @@ sqlite3
393393
:class:`sqlite3.Connection` for creating aggregate window functions.
394394
(Contributed by Erlend E. Aasland in :issue:`34916`.)
395395

396+
* Add :meth:`~sqlite3.Connection.blobopen` to :class:`sqlite3.Connection`.
397+
:class:`sqlite3.Blob` allows incremental I/O operations on blobs.
398+
(Contributed by Aviv Palivoda and Erlend E. Aasland in :issue:`24905`)
399+
396400

397401
sys
398402
---

‎Lib/test/test_sqlite3/test_dbapi.py

Copy file name to clipboardExpand all lines: Lib/test/test_sqlite3/test_dbapi.py
+156-1Lines changed: 156 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@
3333
check_disallow_instantiation,
3434
threading_helper,
3535
)
36+
from _testcapi import INT_MAX
37+
from os import SEEK_SET, SEEK_CUR, SEEK_END
3638
from test.support.os_helper import TESTFN, unlink, temp_dir
3739

3840

@@ -1041,11 +1043,163 @@ def test_same_query_in_multiple_cursors(self):
10411043
self.assertEqual(cu.fetchall(), [(1,)])
10421044

10431045

1046+
class BlobTests(unittest.TestCase):
1047+
def setUp(self):
1048+
self.cx = sqlite.connect(":memory:")
1049+
self.cx.execute("create table test(b blob)")
1050+
self.data = b"this blob data string is exactly fifty bytes long!"
1051+
self.cx.execute("insert into test(b) values (?)", (self.data,))
1052+
self.blob = self.cx.blobopen("test", "b", 1)
1053+
1054+
def tearDown(self):
1055+
self.blob.close()
1056+
self.cx.close()
1057+
1058+
def test_blob_seek_and_tell(self):
1059+
self.blob.seek(10)
1060+
self.assertEqual(self.blob.tell(), 10)
1061+
1062+
self.blob.seek(10, SEEK_SET)
1063+
self.assertEqual(self.blob.tell(), 10)
1064+
1065+
self.blob.seek(10, SEEK_CUR)
1066+
self.assertEqual(self.blob.tell(), 20)
1067+
1068+
self.blob.seek(-10, SEEK_END)
1069+
self.assertEqual(self.blob.tell(), 40)
1070+
1071+
def test_blob_seek_error(self):
1072+
msg_oor = "offset out of blob range"
1073+
msg_orig = "'origin' should be os.SEEK_SET, os.SEEK_CUR, or os.SEEK_END"
1074+
msg_of = "seek offset results in overflow"
1075+
1076+
dataset = (
1077+
(ValueError, msg_oor, lambda: self.blob.seek(1000)),
1078+
(ValueError, msg_oor, lambda: self.blob.seek(-10)),
1079+
(ValueError, msg_orig, lambda: self.blob.seek(10, -1)),
1080+
(ValueError, msg_orig, lambda: self.blob.seek(10, 3)),
1081+
)
1082+
for exc, msg, fn in dataset:
1083+
with self.subTest(exc=exc, msg=msg, fn=fn):
1084+
self.assertRaisesRegex(exc, msg, fn)
1085+
1086+
# Force overflow errors
1087+
self.blob.seek(1, SEEK_SET)
1088+
with self.assertRaisesRegex(OverflowError, msg_of):
1089+
self.blob.seek(INT_MAX, SEEK_CUR)
1090+
with self.assertRaisesRegex(OverflowError, msg_of):
1091+
self.blob.seek(INT_MAX, SEEK_END)
1092+
1093+
def test_blob_read(self):
1094+
buf = self.blob.read()
1095+
self.assertEqual(buf, self.data)
1096+
1097+
def test_blob_read_oversized(self):
1098+
buf = self.blob.read(len(self.data) * 2)
1099+
self.assertEqual(buf, self.data)
1100+
1101+
def test_blob_read_advance_offset(self):
1102+
n = 10
1103+
buf = self.blob.read(n)
1104+
self.assertEqual(buf, self.data[:n])
1105+
self.assertEqual(self.blob.tell(), n)
1106+
1107+
def test_blob_read_at_offset(self):
1108+
self.blob.seek(10)
1109+
self.assertEqual(self.blob.read(10), self.data[10:20])
1110+
1111+
def test_blob_read_error_row_changed(self):
1112+
self.cx.execute("update test set b='aaaa' where rowid=1")
1113+
with self.assertRaises(sqlite.OperationalError):
1114+
self.blob.read()
1115+
1116+
def test_blob_write(self):
1117+
new_data = b"new data".ljust(50)
1118+
self.blob.write(new_data)
1119+
row = self.cx.execute("select b from test").fetchone()
1120+
self.assertEqual(row[0], new_data)
1121+
1122+
def test_blob_write_at_offset(self):
1123+
new_data = b"c" * 25
1124+
self.blob.seek(25)
1125+
self.blob.write(new_data)
1126+
row = self.cx.execute("select b from test").fetchone()
1127+
self.assertEqual(row[0], self.data[:25] + new_data)
1128+
1129+
def test_blob_write_advance_offset(self):
1130+
self.blob.write(b"d"*10)
1131+
self.assertEqual(self.blob.tell(), 10)
1132+
1133+
def test_blob_write_error_length(self):
1134+
with self.assertRaisesRegex(ValueError, "data longer than blob"):
1135+
self.blob.write(b"a" * 1000)
1136+
1137+
def test_blob_write_error_row_changed(self):
1138+
self.cx.execute("update test set b='aaaa' where rowid=1")
1139+
with self.assertRaises(sqlite.OperationalError):
1140+
self.blob.write(b"aaa")
1141+
1142+
def test_blob_write_error_readonly(self):
1143+
ro_blob = self.cx.blobopen("test", "b", 1, readonly=True)
1144+
with self.assertRaisesRegex(sqlite.OperationalError, "readonly"):
1145+
ro_blob.write(b"aaa")
1146+
ro_blob.close()
1147+
1148+
def test_blob_open_error(self):
1149+
dataset = (
1150+
(("test", "b", 1), {"name": "notexisting"}),
1151+
(("notexisting", "b", 1), {}),
1152+
(("test", "notexisting", 1), {}),
1153+
(("test", "b", 2), {}),
1154+
)
1155+
regex = "no such"
1156+
for args, kwds in dataset:
1157+
with self.subTest(args=args, kwds=kwds):
1158+
with self.assertRaisesRegex(sqlite.OperationalError, regex):
1159+
self.cx.blobopen(*args, **kwds)
1160+
1161+
def test_blob_sequence_not_supported(self):
1162+
with self.assertRaises(TypeError):
1163+
self.blob + self.blob
1164+
with self.assertRaises(TypeError):
1165+
self.blob * 5
1166+
with self.assertRaises(TypeError):
1167+
b"a" in self.blob
1168+
1169+
def test_blob_closed(self):
1170+
with memory_database() as cx:
1171+
cx.execute("create table test(b blob)")
1172+
cx.execute("insert into test values (zeroblob(100))")
1173+
blob = cx.blobopen("test", "b", 1)
1174+
blob.close()
1175+
1176+
msg = "Cannot operate on a closed blob"
1177+
with self.assertRaisesRegex(sqlite.ProgrammingError, msg):
1178+
blob.read()
1179+
with self.assertRaisesRegex(sqlite.ProgrammingError, msg):
1180+
blob.write(b"")
1181+
with self.assertRaisesRegex(sqlite.ProgrammingError, msg):
1182+
blob.seek(0)
1183+
with self.assertRaisesRegex(sqlite.ProgrammingError, msg):
1184+
blob.tell()
1185+
1186+
def test_blob_closed_db_read(self):
1187+
with memory_database() as cx:
1188+
cx.execute("create table test(b blob)")
1189+
cx.execute("insert into test(b) values (zeroblob(100))")
1190+
blob = cx.blobopen("test", "b", 1)
1191+
cx.close()
1192+
self.assertRaisesRegex(sqlite.ProgrammingError,
1193+
"Cannot operate on a closed database",
1194+
blob.read)
1195+
1196+
10441197
class ThreadTests(unittest.TestCase):
10451198
def setUp(self):
10461199
self.con = sqlite.connect(":memory:")
10471200
self.cur = self.con.cursor()
1048-
self.cur.execute("create table test(name text)")
1201+
self.cur.execute("create table test(name text, b blob)")
1202+
self.cur.execute("insert into test values('blob', zeroblob(1))")
10491203

10501204
def tearDown(self):
10511205
self.cur.close()
@@ -1080,6 +1234,7 @@ def test_check_connection_thread(self):
10801234
lambda: self.con.create_collation("foo", None),
10811235
lambda: self.con.setlimit(sqlite.SQLITE_LIMIT_LENGTH, -1),
10821236
lambda: self.con.getlimit(sqlite.SQLITE_LIMIT_LENGTH),
1237+
lambda: self.con.blobopen("test", "b", 1),
10831238
]
10841239
if hasattr(sqlite.Connection, "serialize"):
10851240
fns.append(lambda: self.con.serialize())
+3Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Add :meth:`~sqlite3.Connection.blobopen` to :class:`sqlite3.Connection`.
2+
:class:`sqlite3.Blob` allows incremental I/O operations on blobs.
3+
Patch by Aviv Palivoda and Erlend E. Aasland.

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.