Hi, I have problems with migration. My current android sqlcipher version is 3.5.3 and sqlcipher is 3.4.0.
In 3.5.4 you added fix for null chars. But there are other changes in modified UTF-8 :
1.The null character ‘\u0000’ and characters in the range ‘\u0080’ to ‘\u07FF’ are represented by a pair of bytes
2.char values in the range ‘\u0800’ to ‘\uFFFF’ are represented by three bytes https://docs.oracle.com/javase/6/docs/api/java/io/DataInput.html#modified-utf-8
My password doesn’t contain null chars but has a lot of other characters from these ranges
But your containsNull() validation doesn’t give me ability to use modified UTF-8
Hi, my password is
char[] password = new char[]{‘矫’, ‘黴’, ‘䪈’, ‘詧’, ‘ᢧ’, ‘踚’, ‘踚’, ‘\uF149’, ‘庥’, ‘ꉁ’, ‘鏋’, ‘⠉’, ‘쪡’, ‘ᶲ’, ‘\uF5E1’, ‘旚’, ‘탐’, ‘癗’, ‘\uED6D’, ‘䦎’, ‘뚡’, ‘ଜ’, ‘쪞’, ‘\uF0C7’, ‘㜔’, ‘\uF492’, ‘꾞’, ‘樞’, ‘璌’, ‘茮’ ,‘镲’, ‘陸’};
I started “Migrate from modified UTF-8” test and it was failed
06-04 20:59:07.363 10999-10999/net.zetetic.sqlcipher.test I/TestSuiteActivity: Migrate from modified UTF-8 to UTF-8 with null test - success:false
06-04 20:59:07.630 10999-11026/net.zetetic.sqlcipher.test E/Database: file is not a database: , while compiling: select count(*) from sqlite_master;
Hello @Alexei_Ilmovsky In order to look into this any further, we’re going to need you to provide an example database (with any sensitive data removed) that demonstrates this problem. It should be encrypted with a key that works under 3.5.3, but does not work under 3.5.9.
Also, we’ve reviewed your original post, and there might be some misunderstandings as well. Note that the encoding of characters in the range ‘\u0080’ to ‘\u07FF’ is actually the same as with standard UTF-8 (compare the encoding on the page you linked to here: https://en.wikipedia.org/wiki/UTF-8#Description).
Finally, can you explain the origin of the ‘\uF5E1’, ‘\uED6D’, ‘\uF0C7’, and ‘\uF492’? A quick lookup showed that these are in private use areas and aren’t picked up as valid unicode characters (i.e round trip decoding/encoding produces invalid results).
we have array of bytes and convert it to char array. It works good on 3.5.3 and 3.5.9 if we created db there and use it. But we can’t migrate from 3.5.3 to 3.5.4 and upper
P.s: I tried to add my db but I got “Sorry, new users can not upload attachments.”. You can find it here fortestsqlcipher1.zip - Google Drive
And I think you can create your own db with current password on version 3.5.3 if it’s necessary
We have verified on several machines that we are able to open the supplied database with the provided password on both the SQLCipher command line shell, and within the current SQLCipher for Android library without issue. In general, you cannot take random bytes and convert them into a char[] and always expect to get a valid encoding. For now we will consider this issue closed. If in the future you run into another issue, we would ask that you submit a complete test including all code and dependent tests for independent verification.
Its strange, because I get error when I start your test for this db with current password
Possibly you can add ability to set array of bytes in sqlcipher externally. I think it will close a lot of questions
Hello @Alexei_Ilmovsky - that is exactly what we tried, substituting the database you provided along with the key, and it does work. We also independently verified that it is possible to open the database using the command line tool with the same key using the latest SQLCipher.
There is already a way to specific an array of bytes for key material to SQLCipher. This is known as a raw key and is used to provide a string of random data to the library to use directly (thus bypassing key derivation).
But db has to be opened to use this method. But I can’t open it with new version. I would like to have ability to pass array of bytes into getWritableDatabase/getReadableDatabase methods
And I will check all passwords for sqlcipher we has in our app. Possibly I will find something else
Hello @Alexei_Ilmovsky - I’m not suggesting that you use the raw key approach to work around the problem here, rather, to point out that there is a well-established way to use random bytes as a key to SQLCipher that has been in place for many years (without involving unsafe conversations between byte arrays and chars). This approach can be used in any version of SQLCipher.
With regard to this issue, we have tried repeatedly to reproduce the problem using the database and key you provided. We have not observed any errors opening a database. You can help us look into this further by providing a test case that can reliably reproduce the problem:
Adjust the test suite to package a database that you created with a password like you provided in this thread
Construct a test that demonstrates that it is possible to open the database (i.e. test passes) using SQLCipher 3.5.3
Verify that the test fails using 3.5.9
Publish the changes to your own fork on GitHub with a license statement in the test disclaiming copyright (e.g. placing in the public domain) or Licensing it under the same terms as SQLCipher
Hi, you was right. Table I was shared you works good on all sqlcipher versions. But I found passwords I have problems with
I added project with tests to https://github.com/Ilmovsky/SQLTest. You can find there 2 tests for creation new table and checking migration. If you start migration test it will work good with 3.5.3 but if you update version on gradle to 3.5.4 or upper this test will be failed.
Also you can create new table on version 3.5.3 using create table test. Then you should put it in assets. It wont work with 3.5.4 and above too
Thank you for your replication example, we have been able to verify the behavior you are reporting. The key provided in the example doesn’t appear to be valid modified UTF-8, so there doesn’t seem to be a predictable way to determine whether to invoke the conversion. Since you are utilizing the Community edition of SQLCipher for Android, we feel the most expedient approach would be for you to repackage the library source, changing this line to always being invoked. The cost of always invoking a secondary keying operation on failed password attempts, rather than gating the operation aren’t viable for the general purpose library. We did test this change internally and were able to successfully migrate your database to use a UTF-8 encoded key.
Thanks for answer
Possibly could you add ability to turn on/off this containsNull() validation externally because it will solve big count of problems binding with necessity repackaging the library source code every time version of library will be increased
i think it shouldn’t take a lot of time, because you only need to override getWritableDatabase and getReadableDatabase with ability set there parameter which let developer reject this validation
And thanks a lot for your help
We do not have any plans to allow external control over the containsNull check within SQLCipher for Android. The modification for your needs are small and isolated from the general user base of SQLCipher for Android. We provide build instructions for the library, which can then be integrated as a local AAR package within your application.
