The SQLCipher for Android project provides access to the native SQLCipher APIs via Java-JNI layer which seems to be forked off an early version of Android (2.3.7 perhaps). There were a number of reported issues with this version of Android (sample discussion here).
These SQLite issues seem to have been patched in newer versions of Android. Additionally, the design has changed to allow newer SQLite capabilities such as Write-Ahead-Logging. SQLCipher supports this in the native layers (i.e. iOS/*nix and Win desktop can use) but it cannot be used safely without concurrency issues on Android due to the issues in the Java layer.
Are there any plans from the SQLCipher team to update the Java-JNI layer for Android?
SQLCipher for Android can use the write ahead log, however as you noted above, the Java API is different than that of the current upstream Google android.database.sqlite package. In the case of using write ahead log, you can execute the following PRAGMA statement to enable it:
PRAGMA journal_mode=wal;
We have discussed this in the past, and while we are open to refreshing the API, we still favor being able to support Android clients down to 2.1 currently. Updating the Java/JNI layer itself will be a time consuming endeavor. Is there a specific feature that you find missing, or relevant issues with the current code base?
It was built using the latest version of Android at the time, which is now a while ago. I think upgrading would be useful. We haven’t been having issues with it, but we don’t do heavily concurrency in our apps.
I think the best way to update android-database-sqlcipher would be to submit pull requests with updated code that addressed specific issues. Like @developernotes said, a full update would be a large project. And, “if it ain’t broke, don’t fix it”
Thank you @eighthave@developernotes! Our sql use involves heavy concurrency and have had to resort to application level locking to workaround aforementioned Android issues with concurrent access. The current version of Android has fixed these concurrency issues by using a completely new architecture. By allowing multi-threaded access of the database using WAL, we could also have possible performance improvements.
Indeed, the upgrade of this layer is a huge undertaking. I can submit a PR for this, but it would be a huge patch. Would this be something of interest to you?
From my point of view, the update process would be easier if submitted in as small chunks as possible. But Zetetic is doing almost all of the work on android-database-sqlcipher these days, so I’d defer to them on how they want to handle it.
I should add that we did (Guardian Project with Zetetic’s help) some similar work with the IOCipher library, which is also based on SQLCipher. It takes a similar approach to android-database-sqlcipher in that it is built using native Android components, in this case the java.io.File API. In that case, we have it working with concurrency pretty well. So it might provide a useful example.
Thanks @eighthave! Great work on the IOCipher library!
Smaller chunks may be hard to provide considering the pull from AOSP and having it work with SQLCipher native shared libaries would be a big commit. @developernotes Would Zetetic be open to pulling in this update into the android-database-sqlcipher project?
Thank you for your interest in updating SQLCipher for Android! Unfortunately, with such a large pull request, verifying the downstream impact would be a major undertaking and very error prone. It would make auditing nearly impossible. Our initial recommendation would be to isolate a specific issue you have that is not addressed in the current source tree and open an issue so a discussion can be had about the best way to approach a resolution. Thanks!
@developernotes The issue with the current android-database-sqlcipher stems from AOSP issues from the original version that was forked. Those were the reasons Google updated these SQLite access layers to allow for heavy concurrency. Could the risk be mitigated if unit tests were added to the layer?
That is certainly possible, however, producing additional characterization tests to cover the entire codebase would be very time consuming. Is this something that you have already attempted? That said, my preference moving forward still stands in this case, I would rather investigate a specific scenario within a Github issue where we can discuss in detail how it may be best addressed.
We can certainly open an issue. I will request one for enabling WAL layer similar to documented here - Regarding concurrency, it would be hard to provide a reproducibility of the concurrency scenario as others have failed too.
The merits of the proposed change would certainly benefit the community in general as well as bring SQLCipher APIs closer to Android L SQLite.
