net.sqlcipher.database.SQLiteException: file is encrypted or is not a database: , while compiling: select count(*) from sqlite_master;

SQLCipher Issues
1

Version of SQLCipher using in the app : 3.5.4

Difficult part is, i can’t able to reproduce this issue at my side. there are more than 3k issues are reported from the field. our users are really losing data and app is no more useful for them.

After my investigation on the same, i came know that - Snapdragon 800/801 couldn’t match the AES encryption speeds of the ARMv8 chips because it doesn’t have hardware encryption. is this true ?

Somewhere down the line devices are incapable of decrypting data.

Most of the issues are from Samsung devices
:

May i know what could be the issue, What can be done here to fix issue?

Exception java.lang.RuntimeException: Unable to create application : net.sqlcipher.database.SQLiteException: file is encrypted or is not a database: , while compiling: select count() from sqlite_master;
android.app.ActivityThread.handleBindApplication (ActivityThread.java:4775)
android.app.ActivityThread.access$1600 (ActivityThread.java:170)
android.app.ActivityThread$H.handleMessage (ActivityThread.java:1364)
android.os.Handler.dispatchMessage (Handler.java:102)
android.os.Looper.loop (Looper.java:146)
android.app.ActivityThread.main (ActivityThread.java:5635)
java.lang.reflect.Method.invokeNative (Method.java)
java.lang.reflect.Method.invoke (Method.java:515)
com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run (ZygoteInit.java:1291)
com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1107)
dalvik.system.NativeStart.main (NativeStart.java)
arrow_drop_down
Caused by net.sqlcipher.database.SQLiteException: file is encrypted or is not a database: , while compiling: select count() from sqlite_master;
net.sqlcipher.database.SQLiteCompiledSql.native_compile (SourceFile)
net.sqlcipher.database.SQLiteCompiledSql.compile (SourceFile:91)
net.sqlcipher.database.SQLiteCompiledSql. (SourceFile:64)
net.sqlcipher.database.SQLiteProgram. (SourceFile:83)
net.sqlcipher.database.SQLiteQuery. (SourceFile:49)
net.sqlcipher.database.SQLiteDirectCursorDriver.query (SourceFile:42)
net.sqlcipher.database.SQLiteDatabase.rawQueryWithFactory (SourceFile:1787)
net.sqlcipher.database.SQLiteDatabase.rawQuery (SourceFile:1752)
net.sqlcipher.database.SQLiteDatabase.keyDatabase (SourceFile:2427)
net.sqlcipher.database.SQLiteDatabase.openDatabaseInternal (SourceFile:2356)
net.sqlcipher.database.SQLiteDatabase.openDatabase (SourceFile:1116)
net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase (SourceFile:1179)
net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase (SourceFile:162)
net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase (SourceFile:129)
DatabaseModule. (SourceFile:55)
prepareInjectionsGraph (SourceFile:146)
LumeaApplication.onCreate (SourceFile:127)
android.app.Instrumentation.callApplicationOnCreate (Instrumentation.java:1013)
android.app.ActivityThread.handleBindApplication (ActivityThread.java:4772)
android.app.ActivityThread.access$1600 (ActivityThread.java:170)
android.app.ActivityThread$H.handleMessage (ActivityThread.java:1364)
android.os.Handler.dispatchMessage (Handler.java:102)
android.os.Looper.loop (Looper.java:146)
android.app.ActivityThread.main (ActivityThread.java:5635)
java.lang.reflect.Method.invokeNative (Method.java)
java.lang.reflect.Method.invoke (Method.java:515)
com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run (ZygoteInit.java:1291)
com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1107)
dalvik.system.NativeStart.main (NativeStart.java)

2

@chethan_HB typically that error means that the provided key material is incorrect to open the database. Can you provide some information on how the key material is obtained/generated for the database?

3

Hi Sjlombardo,

Thanks for your reply, below details might give you idea on what i am doing here.

These are steps what it goes :

1 . Basically i considered “Master key” as “system current time”

  System.currentTimeMillis();

Because there could be chances of retrieving master key by reverse engineering process if in case hardcoded value kept in the app to be a master key, so we followed approach of considering current time as a master key.

  1. “Master key” is shaving in shared preference with AES encryption :
    below code illustrate storing “Master key” with AES encryption.

    public synchronized boolean storeValueForKey(String userKey, String valueToBeEncrypted, SecureStorageError 
secureStorageError) {
 boolean returnResult = true;
 String encryptedString = null;

 try {
     if(null == userKey || userKey.isEmpty() || userKey.trim().isEmpty() || null == valueToBeEncrypted) {
         secureStorageError.setErrorCode(secureStorageError.UnknownKey);
         returnResult = false;
         return false;
     }

     SecretKey e = this.generateAESKey();
     SecretKeySpec key = new SecretKeySpec(e.getEncoded(), "AES");
     Cipher cipher = Cipher.getInstance("AES/CTR/NoPadding");
     byte[] ivBlockSize = new byte[cipher.getBlockSize()];
     IvParameterSpec ivParameterSpec = new IvParameterSpec(ivBlockSize);
     cipher.init(1, key, ivParameterSpec);
     byte[] encText = cipher.doFinal(valueToBeEncrypted.getBytes());
     encryptedString = Base64.encodeToString(encText, 0);
     returnResult = this.storeEncryptedData(userKey, encryptedString, "App.Storage.file");
     if(returnResult) {
         returnResult = this.storeKey(userKey, e, "AppInfra.Storage.kfile");
         if(!returnResult) {
             this.deleteEncryptedData(userKey, "AppInfra.Storage.file");
         }
     }

     if(!returnResult) {
         secureStorageError.setErrorCode(secureStorageError.StoreError);
     }

     encryptedString = returnResult?encryptedString:null;
     boolean isDebuggable = 0 != (this.mContext.getApplicationInfo().flags & 2);
     if(isDebuggable) {
         this.mAppInfra.getAppInfraLogInstance().log(LogLevel.DEBUG, "Encrypted Data", encryptedString);
     }
 } catch (Exception var13) {
     secureStorageError.setErrorCode(secureStorageError.EncryptionError);
     returnResult = false;
     this.mAppInfra.getAppInfraLogInstance().log(LogLevel.ERROR, "SecureStorage", var13.getMessage());
 }

 return returnResult;

    }
    }

3.The same key will be retrieved from shared preference with decrypted master key to pass sqlcipher to decrypt database. below is the function for the same -

    public synchronized String fetchValueForKey(String userKey, SecureStorageError secureStorageError) {
    String decryptedString = null;
    if(null != userKey && !userKey.isEmpty()) {
        String encryptedString = this.fetchEncryptedData(userKey, secureStorageError, "AppInfra.Storage.file");
        String encryptedAESString = this.fetchEncryptedData(userKey, secureStorageError, "AppInfra.Storage.kfile");
        if(null != encryptedString && null != encryptedAESString) {
            try {
                Key e = this.fetchKey(encryptedAESString, secureStorageError);
                Cipher cipher2 = Cipher.getInstance("AES/CTR/NoPadding");
                byte[] ivBlockSize = new byte[cipher2.getBlockSize()];
                IvParameterSpec ivParameterSpec = new IvParameterSpec(ivBlockSize);
                cipher2.init(2, e, ivParameterSpec);
                byte[] encryptedValueBytes = Base64.decode(encryptedString, 0);
                byte[] decText = cipher2.doFinal(encryptedValueBytes);
                decryptedString = new String(decText);
            } catch (Exception var12) {
                this.mAppInfra.getAppInfraLogInstance().log(LogLevel.ERROR, "SecureStorage", var12.getMessage());
                secureStorageError.setErrorCode(secureStorageError.DecryptionError);
                if(null != decryptedString) {
                    decryptedString = null;
                }
            }

            return decryptedString;
        } else {
            secureStorageError.setErrorCode(secureStorageError.UnknownKey);
            return null;
        }
    } else {
        secureStorageError.setErrorCode(secureStorageError.UnknownKey);
        return null;
    }
}

But How could some thing working all the time, breaks in some point of time and doesn’t recover from that point. It’s seems like master key is corrupted.

This is a hard problem we don’t have a great answer for yet.Troubleshooting this errors is a tricky business without reproducible steps . So a lot of due diligence is needed upfront to even get started with the guesswork.

Could you please propose/suggestion on what we can do here. I hope above details sufficient to share your views.

4

Any update to share ?

5

Hello @chethan_HB - there are a number of issues here, including the use of uninitialized IVs, a low entropy key (system time), etc. In addition like the example you provided is quite incomplete, as we’re unable to see the generation of the wrapper key via this.generateAESKey(), the storage of encrypted data via storeEncryptedData(), the implementation of storeKey, etc.

Frankly, since you’re apparently doing some pretty complicated work storing master keys, etc, there is a high likelihood that there is some problem retrieving the SQLCipher key material, resulting in the error you are seeing. Having a standalone application that demonstrates the problem would be helpful.

6

Thanks for this information, yes even my suspect is also on master key, master key retrieval being failed here in one of the case. I am not getting into this error. but field users are having this issues.

here is the function which your looking for -

  private SecretKey generateAESKey() throws NoSuchAlgorithmException {
    boolean outputKeyLength = true;
    SecureRandom secureRandom = new SecureRandom();
    KeyGenerator keyGenerator = KeyGenerator.getInstance("AES");
    keyGenerator.init(256, secureRandom);
    SecretKey key = keyGenerator.generateKey();
    return key;
}

Any hint for further investigation would be appreciable ?

7

Hello @chethan_HB - I’ve reviewed this thread a few time and unfortunately it fundamentally doesn’t make any sense. The method storeValueForKey generates a random AES key, and then uses it to encrypt valueToBeEncrypted. Then it goes on to store userKey, a value which was not used to encrypt valueToBeEncrypted. The actual key used to encrypt the data (SecretKeySpec key) appears to be discarded. Similarly, in fetchValueForKey, the application seems to load the key via fetchKey (ostensibly this is the stored userKey from earlier?), then use it to decrypt the other information. However, the encrypted string would not have been encrypted under userKey, it would have been encryped under the discarded random AES key. In addition, there are other issues, e.g.

  1. IVs are not properly initialized with random or single use data, a prequisite for using AES/CTR securely
  2. There is no padding, despite the use of variable length input
  3. The apparent use of a low entropy key like System.currentTimeMillis()

Finally, pertinent parts of the code path are still missing, including the logic used to actually store the data (e.g. implementations of storeEncryptedData, storeKey, etc.), the actual use of System.currentTimeMillis().

As a result, I don’t think there is anything further we can do the help you here. There is no evidence from what you’ve provided that this is an issue with SQLCipher, or with any specific devices. Of course, if you can fix some of the issues described here and provide a complete, standalone, reproducible test case that demonstrates a problem with SQLCipher we’d be happy to look into it further.

8

Dear sjLombardo, how do you recommend us to keep the master key value in android app ?. i don’t think it should not be either in shared preference (Persistence storage) or External file. what is the best approach to secure master key in android app .

9

@chethan_HB Key management is a very nuanced topic where the options and security trade offs depend on the security profile, threat model, and design of the application.

For the greatest security, we typically recommend that at least a substantial part of the key material for a SQLCipher database should come directly from the user, and not be stored on the device itself (e.g. a passphrase). It may be possible to use some key material from the device itself, or from a remote service (e.g. this might be possible if a user has to “log in” to a web account to access the app).

Other workable approaches involve using an OS-provided key storage mechanism, e.g. iOS Keychain, Processor Secure Enclave, Android Key Store, etc.

Note that we do not recommend hard-coding a key in application code; it is not suitable for a secure implementation.

10

Dear Team,

Issue is reopened again from the filed, there are many fatal error logged from the field user device, we are not able reproduce this issue internally. it is happening for field users. added crash logs for your review. for the same crash, last time you said it is due to incorrect passphrase value. we have saved passphrase value in persistence storage in the app, there is no way that it goes with incorrect passphrase value. SQL cipher version used in the app is 3.5.4. what might have caused this issue ?

Exception java.lang.RuntimeException: Unable to create application
net.sqlcipher.database.SQLiteException: file is encrypted or is not a database: , while compiling: select count() from sqlite_master;
android.app.ActivityThread.handleBindApplication (ActivityThread.java:5888)
android.app.ActivityThread.-wrap3 (ActivityThread.java)
android.app.ActivityThread$H.handleMessage (ActivityThread.java:1703)
android.os.Handler.dispatchMessage (Handler.java:102)
android.os.Looper.loop (Looper.java:154)
android.app.ActivityThread.mapin (ActivityThread.java:6692)
java.lang.reflect.Method.invoke (Method.java)
com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run (ZygoteInit.java:1468)
com.android.internal.os.ZygoteInit.main (ZygoteInit.java:1358)
arrow_drop_down
Caused by net.sqlcipher.database.SQLiteException: file is encrypted or is not a database: , while compiling: select count() from sqlite_master;
net.sqlcipher.database.SQLiteCompiledSql.native_compile (SQLiteCompiledSql.java)
net.sqlcipher.database.SQLiteCompiledSql.compile (SQLiteCompiledSql.java:91)
net.sqlcipher.database.SQLiteCompiledSql. (SQLiteCompiledSql.java:64)
net.sqlcipher.database.SQLiteProgram. (SQLiteProgram.java:83)
net.sqlcipher.database.SQLiteQuery. (SQLiteQuery.java:49)
net.sqlcipher.database.SQLiteDirectCursorDriver.query (SQLiteDirectCursorDriver.java:42)
net.sqlcipher.database.SQLiteDatabase.rawQueryWithFactory (SQLiteDatabase.java:1787)
net.sqlcipher.database.SQLiteDatabase.rawQuery (SQLiteDatabase.java:1752)
net.sqlcipher.database.SQLiteDatabase.keyDatabase (SQLiteDatabase.java:2427)
net.sqlcipher.database.SQLiteDatabase.openDatabaseInternal (SQLiteDatabase.java:2356)
net.sqlcipher.database.SQLiteDatabase.openDatabase (SQLiteDatabase.java:1116)
net.sqlcipher.database.SQLiteDatabase.openOrCreateDatabase (SQLiteDatabase.java:1179)
net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase (SQLiteOpenHelper.java:162)
net.sqlcipher.database.SQLiteOpenHelper.getWritableDatabase (SQLiteOpenHelper.java:129)

11

Hello @chethan_HB - unfortunately there isn’t enough information to go on here. Regardless, as a first step, we would recommend upgrading to the latest version of SQLCipher. There are several things that could be causing the problem, e.g. application logic errors handling the key and initializing SQLCipher (i.e. getting the data out of the preference). Other more remote possibilities include the preferences storing the key material being cleared or corrupted, or an issue with the database file becoming corrupted, etc. However, without some steps to reproduce or an actual completely working reference application showing the code you are using, it is impossible to speculate.

12

Below is my stacktrace:

at net.sqlcipher.database.SQLiteCompiledSql.native_compile (SQLiteCompiledSql.java)
at net.sqlcipher.database.SQLiteCompiledSql.compile (SQLiteCompiledSql.java:91)
at net.sqlcipher.database.SQLiteCompiledSql. (SQLiteCompiledSql.java:64)
at net.sqlcipher.database.SQLiteProgram. (SQLiteProgram.java:83)
at net.sqlcipher.database.SQLiteQuery. (SQLiteQuery.java:49)
at net.sqlcipher.database.SQLiteDirectCursorDriver.query (SQLiteDirectCursorDriver.java:42)
at net.sqlcipher.database.SQLiteDatabase.rawQueryWithFactory (SQLiteDatabase.java:1787)
at net.sqlcipher.database.SQLiteDatabase.rawQuery (SQLiteDatabase.java:1752)
at net.sqlcipher.database.SQLiteDatabase.keyDatabase (SQLiteDatabase.java:2427)
at net.sqlcipher.database.SQLiteDatabase.openDatabaseInternal (SQLiteDatabase.java:2356)
at net.sqlcipher.database.SQLiteDatabase.openDatabase (SQLiteDatabase.java:1116)
at net.sqlcipher.database.SQLiteDatabase.openDatabase (SQLiteDatabase.java:1008)
at net.sqlcipher.database.SQLiteOpenHelper.getReadableDatabase (SQLiteOpenHelper.java:249)
at net.sqlcipher.database.SQLiteOpenHelper.getReadableDatabase (SQLiteOpenHelper.java:214)

I have seen many replies to similar question and all of them suggesting the same thing that the KEY you are using to decrypt the database is different than what you used to encrypt in the first place.

I was wondering if this could be one an issue with the manifest tag “allowBackup: true”. I have seen cases where developers are saying that even after installing/uninstalling/installing app, the previous data shows up in the data/data/com.packagename folder.

Could this be an issue, as the app is trying to decrypt a database that was created from the previous release which had a different key.

Could “allowBackup: true” be the actual culprit?

13

Hi @Shishir_Shetty

The key used in SQLCipher to encrypt and decrypt data does not change for a given release of the library, once a database is keyed, the existing password is required to rekey the database. However if a user supplied an invalid password, for example an older backup database that was restored which was keyed with a different password that what they expected (possibly through changing the password via the keying API but was not backed up), the library would not be able to decrypt the database. The user would need to provide the password that was used most recently relative to the backup to decrypt the database.

14

Hey @sjlombardo

I’m using application which uses sqlcipher and it uses iOS keychain, instead of enclave. Apologies for my ignorance, but would the sqlcipher API lend itself into using enclave?

As far as I understand, sqlcipher wouldn’t do any of the decryption/encryption, it would just send encrypted data to enclave and get decrypted back.

Now, if API doesn’t lend itself to that, then I guess app makers would have butcher it in, and I feel like lot of them might make security omissions there zetetic would not make.

15

Hello @ytti - SQLCipher wouldn’t operate in the manner you describe where the enclave is doing the encryption and decryption. However, it is certainly possible to use the iOS Keychain / enclave to store an encryption key for SQLCipher. In fact, many applications operate this way, especially to facilitate use of biometrics. For example, if an application stores the SQLCipher key material that way, it can facilitate unlocking a database using TouchID or FaceID.

16

Just for my understanding, can you explain roughly what the process would then be.

I’m guessing, SQLCipher gives password protected secret key to enclave and enclave would return unlocked secret key, and SQLCipher would then proceed to decrypt the data?

So if there would be some malicious App and OS bug allowing cross PID memory access, does this mean the malicious app could get copy of the secret key?

I guess it doesn’t fundamentally matter much, because in scenario where SQLCipher would give encrypted data and password and get decrypted data from enclave, same attacker could read the password and later ask enclave to decrypt what it wants?

Or are there differences in attack surface in these two methods? Are there some BCP document how this should be used?
Is there some actions to reduce the attack surface like keeping the secret key in memory in readable manner for minimum amount of clocks?

17

Hello @ytti - The scenario you are describing is closer to how something like intel SGX would work. The iOS Enclave/Keychain is not that advanced, nor is the Andriod Keystore. In any case the Application is always responsible for providing the key material to SQLCipher. The library does not integrate directly, the application needs to bridge that gap. As you’ve noted, there is very little difference in the attack scenarios either way.

Once the Application obtains the key material, it hands it off to SQLCipher which would then derive the actual encryption key, and finally use it to secure the data in the database file.

With respect to a hypothetical cross PID memory access bug, yes, that would allow a malicious app to get the key. However, it is exceptionally difficult for SQLCipher to defend against a scenario where the attacker has direct memory access.

An application can certainly reduce the time that the key is in memory by opening and then closing the database handle immediately (which would wipe the key). Note however that this will have it’s own trade offs in terms of performance.