Android API Update

SQLCipher
1

There have been several discussions in years past about updating the SQLCipher Android API to be more in line with the latest platform SQLite implementation. There have been some updates in the recent releases to add new API methods, which have been appreciated, but the code in SQLiteDatabase remains largely unchanged from the Android Gingerbread implementation. Is it on your road map to revamp this API, bringing the performance improvements of the latest platform SQLiteDatabase to SQLCipher? Considering most of these changes were introduced in Jelly Bean API 16 and 99.5% of Android devices are running Jelly Bean or newer, it would be a much welcomed update.

Specifically I’m interested in improved database concurrency, as this has been a consistent problem in our app using SQLCipher. The current implementation locks the database on nearly all accesses, transactions, writes, and reads (before and during), which prevents SQLite WAL journal mode from allowing multiple readers along with a single writer. Following the locking pattern in the latest platform implementation would be a massive improvement in concurrency performance.

1 Like
2

Hello @Jeff_Lockhart

Thanks for reaching out with regard to the SQLCipher for Android API. We certainly agree that making adjustments to the the locking behavior within the library would be a beneficial approach for all users. While we understand the demographic continues to advance, we have long supported a much wider swath of Android API versions which is a consideration when changes are evaluated and introduce to the library. That said, beyond the concurrency issue, are there specific features that you are looking for not present in the API? This is certainly on our radar, however we do not have a concrete timeline yet.

3

Concurrency performance is the #1 issue we experience using SQLCipher for Android. We see many ANR reports, especially from older devices with slower I/O, that stem from what should be small database reads that occur on the main UI thread, that are blocked on SQLiteDatabase.mLock because a background thread is performing another longer running database operation. SQLite, especially in WAL journal mode, should be able to perform these database operations concurrently, but the SQLiteDatabase API has this internal lock, which Google’s SQLite API has improved the behavior of since the Jelly Bean platform update.

It’s admirable you’ve maintained backwards compatibility support for so many Android APIs. But even the 0.5% of devices that are running an older platform than Jelly Bean, could still run an app using SQLCipher with an updated SQLiteDatabase implementation based on the latest Android platform code. Updating the API would only affect apps that might still be calling one of a few deprecated methods in SQLiteDatabase. But there’s really no reason an app would need to be calling those methods, 7 years after they were deprecated. Every one of the deprecated methods either has a safer replacement, serves no purpose anymore, or specifically is used to deal with the poor locking behavior of the older API.

yieldIfContended()

This method was deprecated in API level 15. if the db is locked more than once (because of nested transactions) then the lock will not be yielded. Use yieldIfContendedSafely instead.

getSyncedTables()

This method was deprecated in API level 15. This method no longer serves any useful purpose and has been deprecated.

markTableSyncable(String table, String deletedTable)

This method was deprecated in API level 15. This method no longer serves any useful purpose and has been deprecated.

markTableSyncable(String table, String foreignKey, String updateTable)

This method was deprecated in API level 15. This method no longer serves any useful purpose and has been deprecated.

setLockingEnabled(boolean lockingEnabled)

This method was deprecated in API level 16. This method now does nothing. Do not use.

isDbLockedByOtherThreads()

This method was deprecated in API level 16. Always returns false. Do not use this method.

There is no longer the concept of a database lock, so this method always returns false.

With the Android platform SQLite code updates, the deprecated methods are still present, so updating the code doesn’t actually break old apps. The methods simply don’t do anything, as they no longer need to. It would work the same updating SQLCipher’s Android API and would bring much improved concurrency performance for all SQLCipher Android apps.

If I had the time to dedicate to it, I would be willing to assist in these updates, certainly with testing.

4

Hello @Jeff_Lockhart

Thank you again for your feedback, it is very helpful to hear from users of the library. We will certainly take these into consideration during our review along with the concurrency issue you raised. We will certainly reach out online when we have a beta available for testing.

5

Much appreciated, thank you. Looking forward to this update.

6

@developernotes Are there any plans in the near future to update the locking mechanisms to be more in line with modern android? Jeff brought up great points about performance, and that’s certainly one huge downside about the current locking structure.

However, this isn’t just a performance problem. The way SQLCipher handles locking makes it very easy to deadlock applications. Because beginTransaction() and endTransaction() are actually acquiring/releasing a re-entrant lock under the hood, it means that it’s very easy to accidentally acquire another lock within a transaction and wind-up in a bad situation.

The most common example I can think of is some caching layer over a database. Say you have a cache around a table, with some interface:

public synchronized MyModel get(long id) {
  // if it's in the cache, return it
  // otherwise fetch from DB, cache it, return it
}

Now, imagine you accessed that in a transaction:

db.beginTransaction();
// ...
myCache.get(id);
// ...
db.endTransaction();

Boom, potential deadlock. It’s possible for Thread A to have the lock on the cache, but unable to get the database lock, because Thread B could have the database lock and is unable to get the lock on the cache.

I have run into several flavors of this issue, and solutions usually revolve around being forced to treat database transactions as your only locking structure, or to do silly dances around accessing other locks inside of transactions.

Android moved away from this current locking approach not just for performance reasons, but for these types of threading issues it would cause. You can see their code comments here:

https://cs.android.com/android/platform/superproject/+/master:frameworks/base/core/java/android/database/sqlite/SQLiteDatabase.java;l=133?q=sqlitedatabase&ss=android

As a follow-up question: are the changes required to update the API just in java-land, or is there some other underlying issue in native code preventing the API change? The javadoc for SQLiteDatabase#lockForced() was a bit concerning.

1 Like
7

Hello @greyson-signal

Thanks for your feedback on the SQLCipher for Android API. We are currently in the process of completing a new SQLCipher release, however, we do plan to review and address the concurrency issues in the Java library (and any potential native underpinnings) following the impending release which will be include in the the following future release.

2 Likes
8

Hello @developernotes

It’s great that you are looking at concurrency issues. Do you have any ideas when a version with improvements will be released?