I am experiencing crashes with SQLCipher 4.5.6 on a Debian 12-based OS using Shared-Cache mode when trying to attach a DB concurrently from multiple threads within our application.
It seems like multiple threads are trying to access the same cipher_ctx object instance that is unfortunately freed by one thread, which results in a use-after-free crash.
My issue is similar to this ticket.
Is Shared-Cache mode supported by SQLCipher? Is it our responsibility to synchronize open/attach operations to encrypted databases? I believe I understand that, once a single connection is established to a database, that connection can be used by multiple threads to read/write to the database, given the correct thread safety compiler flags.
Can you confirm if this is an issue within SQLCipher OR this is a known limitation that should be handled by the integrating application OR this should be possible, but we are using the API incorrectly?
I added a thread ID to the sqlcipher debug logs which highlights the use-after-free really nicely.
I can email the full log if needed (new users cannot attach zip files to tickets).
Details
SQLCipher is compiled into a static library with given compiler flags:
-DSQLITE_ENABLE_COLUMN_METADATA -DSQLITE_ENABLE_UNLOCK_NOTIFY -DSQLITE_MAX_ATTACHED=125 -DSQLITE_ENABLE_MEMORY_MANAGEMENT -DSQLITE_THREADSAFE=2 -DSQLITE_ENABLE_MEMSYS5=1 -DSQLITE_DEFAULT_PAGE_SIZE=8192 -DSQLITE_HAS_CODEC=1 -DSQLITE_TEMP_STORE=2
Callstack
#2 <signal handler called>
#3 0x00007f82084cf850 in sqlcipher_codec_ctx_set_error (error=1, ctx=0x7f81dc27ddc8) at sqlite3.c:108407
#4 sqlite3Codec (iCtx=0x7f81dc27ddc8, data=<optimized out>, pgno=1, mode=3) at sqlite3.c:41565
#5 0x00007f82084912fb in readDbPage (pPg=pPg@entry=0x7f81dc290ae0) at sqlite3.c:59744
#6 0x00007f82084f2b17 in getPageNormal (pPager=0x7f81dc289dc8, pgno=1, ppPage=0x7f81fd147780, flags=<optimized out>) at sqlite3.c:62299
#7 0x00007f82084f705a in sqlite3PagerGet (flags=0, ppPage=0x7f81fd147780, pgno=1, pPager=<optimized out>) at sqlite3.c:62428
#8 btreeGetPage (flags=0, ppPage=<synthetic pointer>, pgno=1, pBt=0x7f81c82723d8) at sqlite3.c:6889
#9 lockBtree (pBt=0x7f81c82723d8) at sqlite3.c:7833
#10 btreeBeginTrans (p=p@entry=0x7f81dc285bf8, wrflag=wrflag@entry=0, pSchemaVersion=pSchemaVersion@entry=0x0) at sqlite3.c:8228
#11 0x00007f8208543fc4 in sqlite3BtreeBeginTrans (pSchemaVersion=<optimized out>, wrflag=<optimized out>, p=<optimized out>) at sqlite3.c:73857
#12 sqlite3InitOne (db=db@entry=0x7f81dc001558, iDb=iDb@entry=2, pzErrMsg=pzErrMsg@entry=0x7f81fd147950, mFlags=mFlags@entry=0) at sqlite3.c:14863
#13 0x00007f82085447f8 in sqlite3Init (db=db@entry=0x7f81dc001558, pzErrMsg=pzErrMsg@entry=0x7f81fd147950) at sqlite3.c:146136
#14 0x00007f8208545834 in attachFunc (context=0x7f81dc00d068, NotUsed=<optimized out>, argv=0x7f81dc00d098) at sqlite3.c:124796
#15 0x00007f820855b072 in sqlite3VdbeExec (p=p@entry=0x7f81dc00a7b8) at sqlite3.c:101251
#16 0x00007f820853ce80 in sqlite3Step (p=0x7f81dc00a7b8) at sqlite3.c:90720
#17 sqlite3_step (pStmt=pStmt@entry=0x7f81dc00a7b8) at sqlite3.c:25245
#18 0x00007f8208540178 in sqlite3_exec (db=<optimized out>, zSql=<optimized out>, xCallback=<optimized out>, pArg=<optimized out>, pzErrMsg=<optimized out>) at sqlite3.c:140223
#19 0x00007f82083f5ce6 in sqlite3_blocking_exec (db=0x7f81dc001558, sql=0x7f81dc26a400 "ATTACH DATABASE 'MyDatabase.db3' AS 'MyDatabase' KEY 'SQLCipherTestPassword123456';", callback=callback@entry=0x0, arg=arg@entry=0x0, errmsg=errmsg@entry=0x7f81fd147e28, pCommand=pCommand@entry=0x0, isReadContext=true)
Logs right before the crash:
Hello @TekMate,
It seems like you may be using the SQLite API incorrectly due to a misunderstanding of the thread safety options.
The compiler flag -DSQLITE_THREADSAFE=2, “Multi-Thread” does not make it safe to use SQLite objects like connections across multiple threads:
Multi-thread. In this mode, SQLite can be safely used by multiple threads provided that no single database connection nor any object derived from database connection, such as a prepared statement, is used in two or more threads at the same time.
In other words, the application is strictly required to synchronize access to all SQLite objects so that only one thread is using any individual object. Thus it is not safe to use multiple threads to read/write to the database.
Given the description of your application you should switch to use Serialized mode (-DSQLITE_THREADSAFE=1):
Serialized. In serialized mode, API calls to affect or use any SQLite database connection or any object derived from such a database connection can be made safely from multiple threads.
That seems like the level of thread safety you are expecting. Serialized is the default mode in official SQLCipher builds and the recommended setting. Please give that a try and let us know if it resolves the issue.
Finally, Shared Cache mode is strongly recommended against, both in SQLite and in SQLCipher. To quote the SQLite Documentation:
Shared-cache mode is an obsolete feature. The use of shared-cache mode is discouraged. Most use cases for shared-cache are better served by WAL mode.
and
Shared cache is disabled by default. It is recommended that it stay that way. In other words, do not use this routine. This interface continues to be provided for historical compatibility, but its use is discouraged. Any use of shared cache is discouraged.
While it is probably unrelated to the issue you reported, you should avoid using the shared cache feature.
Thank you for your reply.
The crash still occurs with -DSQLITE_THREADSAFE=1 and Shared Cache mode.
I created a minimal reproduction of the crash.
I compiled SQLCipher with the same flags, except for -DSQLITE_THREADSAFE=1.
The crash stops happening if we skip sqlite3_enable_shared_cache(1).
The problem is, we are forced to use Shared Cache mode in our application.
Code
#include <mutex>
#include <thread>
#include <iostream>
#include <sqlite3.h>
#include <atomic>
#include <chrono>
using namespace std;
atomic<int> readSignal = 0;
void readThreadFunc(sqlite3 *db) {
sqlite3 *memDb = nullptr;
int res = sqlite3_open(":memory:", &memDb);
if (res != SQLITE_OK) {
cout << "sqlite3_open :memory: failed: " << res << endl;
return;
}
// Wait for the signal to start reading
readSignal.wait(0);
// Attach database
res = sqlite3_exec(memDb, "ATTACH DATABASE 'test_db.db3' AS test_db KEY 'test_key';", nullptr, nullptr, nullptr);
if (res != SQLITE_OK) {
cout << "sqlite3_exec ATTACH failed: " << res << endl;
sqlite3_close(memDb);
return;
}
// Read the only entry in the table
sqlite3_stmt *stmt = nullptr;
res = sqlite3_prepare_v2(memDb, "SELECT value FROM test_db.test WHERE id = 1;", -1, &stmt, nullptr);
if (res != SQLITE_OK) {
cout << "sqlite3_prepare_v2 failed: " << res << endl;
sqlite3_close(memDb);
return;
}
res = sqlite3_step(stmt);
if (res == SQLITE_ROW) {
const char *value = reinterpret_cast<const char *>(sqlite3_column_text(stmt, 0));
cout << "Read value: " << value << endl;
} else if (res == SQLITE_DONE) {
cout << "No data found." << endl;
} else {
cout << "sqlite3_step failed: " << res << endl;
}
// Finalize the statement and close the memory database
sqlite3_finalize(stmt);
sqlite3_close(memDb);
}
int main ()
{
int res = sqlite3_initialize();
if (res != SQLITE_OK) {
cout << "sqlite3_initialize failed: " << res << endl;
return res;
}
res = sqlite3_enable_shared_cache(1);
if (res != SQLITE_OK) {
cout << "sqlite3_enable_shared_cache failed: " << res << endl;
return res;
}
{
sqlite3 *db = nullptr;
res = sqlite3_open("test_db.db3", &db);
if (res != SQLITE_OK) {
cout << "sqlite3_open failed: " << res << endl;
return res;
}
res = sqlite3_key(db, "test_key", 8);
if (res != SQLITE_OK) {
cout << "sqlite3_key failed: " << res << endl;
sqlite3_close(db);
return res;
}
// Create a table and insert some data
res = sqlite3_exec(db, "CREATE TABLE test (id INTEGER PRIMARY KEY, value TEXT);", nullptr, nullptr, nullptr);
if (res != SQLITE_OK) {
cout << "sqlite3_exec failed: " << res << endl;
sqlite3_close(db);
return res;
}
res = sqlite3_exec(db, "INSERT INTO test (value) VALUES ('Hello, World!');", nullptr, nullptr, nullptr);
if (res != SQLITE_OK) {
cout << "sqlite3_exec failed: " << res << endl;
sqlite3_close(db);
return res;
}
sqlite3_close(db);
}
// Start 4 read threads
thread readThreads[4];
for (int i = 0; i < 4; ++i) {
readThreads[i] = thread(readThreadFunc, nullptr);
}
// Give the threads some time to start
this_thread::sleep_for(chrono::milliseconds(500));
// Signal the threads to start reading
readSignal = 1;
readSignal.notify_all();
// Wait for all threads to finish
for (int i = 0; i < 4; ++i) {
readThreads[i].join();
}
return 0;
}
Hello @TekMate - Thanks for trying to provide a minimal reproduction, that is definitely the most helpful thing you can do to track this down.
I was about to write back that I was unable to reproduce with the code, but I was ultimately able to generate a crash with the test program on v4.5.6. It wasn’t 100% reproducible, maybe one of out of 10 or 15 tries.
That said I have been unable to reproduce at all with 4.7.0. Can you please try with the latest release of SQLCipher, since there have been many improvements and fixes between those versions?
Let me know if updating resolves the issue.
The crash still happens with 4.7.0. with the repro app.
In this crash, ctx->write_ctx seems to be an invalid pointer.
Summary
(gdb) where
#0 0x00005555555ee809 in sqlcipher_codec_key_derive (ctx=ctx@entry=0x7ffff76150c8)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:109398
#1 0x00005555555eeb1f in sqlite3Codec (iCtx=0x7ffff76150c8, data=0x7fffe80108e8, pgno=1, mode=3)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:110755
#2 0x0000555555598aab in readDbPage (pPg=pPg@entry=0x7fffe8011920)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:60625
#3 0x00005555555fe997 in getPageNormal (pPager=0x7ffff0010af8, pgno=1, ppPage=0x7ffff6e12900,
flags=<optimized out>)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:63181
#4 0x0000555555602fba in sqlite3PagerGet (flags=0, ppPage=0x7ffff6e12900, pgno=1,
pPager=<optimized out>)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:63310
#5 btreeGetPage (flags=0, ppPage=<synthetic pointer>, pgno=1, pBt=0x7ffff0010628)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:7914
#6 lockBtree (pBt=0x7ffff0010628)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:8858
#7 btreeBeginTrans (p=p@entry=0x7fffe80103a8, wrflag=wrflag@entry=0,
pSchemaVersion=pSchemaVersion@entry=0x0)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:9253
#8 0x0000555555650914 in sqlite3BtreeBeginTrans (pSchemaVersion=<optimized out>,
wrflag=<optimized out>, p=<optimized out>)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:74882
#9 sqlite3InitOne (db=db@entry=0x7fffe8000b78, iDb=iDb@entry=2,
pzErrMsg=pzErrMsg@entry=0x7ffff6e12ac0, mFlags=mFlags@entry=0)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:17756
#10 0x00005555556510c8 in sqlite3Init (db=db@entry=0x7fffe8000b78,
pzErrMsg=pzErrMsg@entry=0x7ffff6e12ac0)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:149022
#11 0x0000555555652685 in attachFunc (context=0x7fffe800db38, NotUsed=<optimized out>,
argv=<optimized out>)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:127181
#12 0x0000555555643e3a in sqlite3VdbeExec (p=p@entry=0x7fffe800ba08)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:102652
#13 0x000055555564eab0 in sqlite3Step (p=0x7fffe800ba08)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:91901
#14 sqlite3_step (pStmt=pStmt@entry=0x7fffe800ba08)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:26426
#15 0x000055555564f9b8 in sqlite3_exec (db=<optimized out>, zSql=<optimized out>,
xCallback=<optimized out>, pArg=<optimized out>, pzErrMsg=<optimized out>)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:142979
#16 0x000055555555fdee in readThreadFunc(sqlite3*) ()
#17 0x00007ffff79ef4a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#18 0x00007ffff77a3044 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007ffff782361c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) frame
#0 0x00005555555ee809 in sqlcipher_codec_key_derive (ctx=ctx@entry=0x7ffff76150c8)
at /root/workspaces/RTIL/src/PLF/ExternLibs/SQLite/V3440200/sqlite3.c:109398
109398 if(ctx->write_ctx->derive_key) {
(gdb) print *ctx
$10 = {store_pass = 0, kdf_iter = 0, fast_kdf_iter = 0, kdf_salt_sz = 0, key_sz = -144617224,
iv_sz = 32767, block_sz = 16, page_sz = 1, reserve_sz = 0, hmac_sz = 0, plaintext_header_sz = 0,
hmac_algorithm = 0, kdf_algorithm = -144617176, error = 32767, flags = 32,
kdf_salt = 0x800000001 <error: Cannot access memory at address 0x800000001>,
hmac_kdf_salt = 0x7ffff7615138 "", buffer = 0x7ffff7615168 "", pBt = 0x7ffff7616308,
read_ctx = 0x7ffff7615158, write_ctx = 0x100000020, provider = 0x0, provider_ctx = 0x0}
(gdb)
Hello @TekMate - We’ve reproduced this issue independently and have a planned update that we believe will resolve the problem. Can you please contact us privately at support@zetetic.net so we can give you a snapshot to test with?
@TekMate - we didn’t hear back from you on this thread or via private channels but we believe this issue should now be resolved in SQLCipher 4.8.0.
