Integrity check incorrect field indexes

[jhmpub]

Hi,

I’m running Codebook 4.5.20.0. I sync via Google Drive between Windows and iOS devices. I periodically perform an integrity check. Today I noticed an issue propagating some new and changed fields. Here are the steps to reproduce:

device #1
File > Integrity Check, No Problems
Sync > Sync with Google Drive
File > Integrity Check, <#> incorrect field indexes fixed
Sync > Sync with Google Drive
File > Integrity Check, No Problems

device #2
File > Integrity Check, No Problems
Sync > Sync with Google Drive
File > Integrity Check, <#> incorrect field indexes fixed
Sync > Sync with Google Drive
File > Integrity Check, No Problems

start over at device #1, ad nauseam, without any additional database changes.

Any idea what’s going on?

[mmoore]

@jhmpub

Thanks for using Codebook and posting to the discussion forum with this question.

The situation you outlined could occur if you edited/added/removed fields on the same Entry on two different devices and then synced.

Here’s a simplified example which can cause this:

Imagine you have an Entry which was already synced on both devices and is fully up to date. It was 3 fields:

Index 1: Username
Index 2: Password
Index 3: Website

One device 1, you add a new Field at the end of the list Note, so the Entry on device 1 looks like this:

Index 1: Username
Index 2: Password
Index 3: Website
Index 4: Note

Without syncing device 1 + device 2 first, on device 2, you add a Note field at the end of the same entry. The entry on device 2 now looks like this:

Index 1: Username
Index 2: Password
Index 3: Website
Index 4: Note

Now you sync device 1, then device 2, then device 1 again to Google Drive. The result is that both devices will have the same data on both side, but the indexes of the newly added Note fields on each side will both be Index 4. So it might look something like this prior to Integrity Check:

Index 1: Username
Index 2: Password
Index 3: Website
Index 4: Note << added by device 1
Index 4: Note << added by device 2

When running Integrity Check, it recognizes that there are incorrect field indexes on this Entry and fixes them up so that they are correctly ascending.

A similar issue could occur if you deleted a field on one device, and prior to syncing it, added/moved other fields around on the same Entry on the other device.

Does this sound similar to the steps you could have taken when this issue occurred?

Typically to avoid this issue, we’d recommend trying to edit an Entry on one device at a time.

This issue should be a lot less likely to occur in Codebook 5 when using Codebook Cloud: Codebook Cloud - Beta Testers Wanted!

[jhmpub]

What you explain is what I would expect with database change collisions.

My problem is I repeatedly get incorrect field indexes fixed after a sync on device 1 and then a sync on device 2 with no database changes.

[mmoore]

@jhmpub

Thanks for the response with the further details.

This does sound like it could be a bug. We’ll work on trying to reproduce it on our end.

One follow up question to help us try to narrow down the reproduction case: if you sync your two devices with no pending changes at all, does it also reproduce the issue? i.e. these steps:

1. Device 1, and Device 2 fully in sync with Google Drive and each other (and Integrity Checks passing).
2. Device 1, sync with Google Drive, Integrity Check << does this produce any Integrity issues?
3. Device 2, sync with Google Drive, Integrity Check << does this produce any Integrity issues?
4. Device 1, sync with Google Drive again, Integrity Check << does this produce any Integrity issues?

Thanks!

[jhmpub]

1. Device 1, and Device 2 fully in sync with Google Drive and each other (and Integrity Checks passing).
check
2. Device 1, sync with Google Drive, Integrity Check << does this produce any Integrity issues?
intermittently, roughly every other time, always 7 incorrect field indexes fixed
3. Device 2, sync with Google Drive, Integrity Check << does this produce any Integrity issues?
intermittently, roughly every other time, always 7 incorrect field indexes fixed
4. Device 1, sync with Google Drive again, Integrity Check << does this produce any Integrity issues?
again, intermittently, roughly every other time, always 7 incorrect field indexes fixed

Hope that helps

[jhmpub]

This integrity check issue turned into a sync failure on ios and windows today.

I got things working again after a database reset:

gdrive
Gear Icon > Settings > Manage apps
Codebook > Options > Delete hidden app data

Windows (primary)
File > Export > Export All
exit Codebook
cd “%APPDATA%\Zetetic LLC”
rmdir /s /q Strip
File > Import
File > Customize Labels
Double click to adjust custom fields: Mode, Masked, etc.
Sync > Setup Sync Key
Sync > Sync with Google Drive

Windows (secondary)
exit Codebook
cd “%APPDATA%\Zetetic LLC”
rmdir /s /q Strip
Sync > Setup Sync Key
Sync > Sync with Google Drive

iOS (secondary)
delete codebook app and reinstall w/ google drive sync
Sync > Setup Sync Key
Sync > Account Authorize
Sync > Sync with Google Drive

[mmoore]

@jhmpub

Thanks for outlining what worked for you. For anyone else looking on, please make sure you have a backup of your Codebook Sync Key before deleting any Codebook data/apps if you’re performing these steps.

One thing to note is that Export/Import doesn’t preserve field ordering so if for example you had an Entry with these fields prior to export/import:

Username: jhmpub
Password: sup3rs3cr3t

After export/import they may be ordered in the opposite direction:

Password: sup3rs3cr3t
Username: jhmpub

I don’t suspect the sync error message is directly related to the field index integrity issues you were getting previously, but they could be. Do you still get the integrity issues after syncing or is that gone?

If this sync error occurs again for you, we’d like to have you collect the sync log from one of your devices and send it over so we can diagnose what caused it. Typically this error message indicates that one of your changesets in Google Drive is corrupt, but the sync log would confirm.

Thanks!

[jhmpub]

@mmoore

I don’t suspect the sync error message is directly related to the field index integrity issues you were getting previously, but they could be. Do you still get the integrity issues after syncing or is that gone?

I was consistently getting 7 incorrect field indexes and the number of entries went down by 7 after the database restore. I have done a lot of syncing between devices and have not seen any field index errors. If the field index error issue is unrelated to the sync issue, it does nonetheless appear the database reset corrected both.

What do I need to do to collect sync logs?

[jhmpub]

This issue is back. Incorrect field indexes fixed after a sync between two systems with no database changes. How do I collect sync logs?

I also noticed entries are not in sync. I’m going to have to reset the database again.

Would a database reset be easier if you brought back the v3.6.x Sync Overwrite and Restore functionality?

[mmoore]

Apologies about missing your previous response asking about how to collect the sync logs. The details are below:

Here are the steps to collect the sync logs:

Windows

Login to Codebook for Windows and select the Help → About menu item located within the main window of the application. Once viewing the About window, click on the Codebook logo 4 times to enable debug mode. Next, close the About window, select the Help → View Log Window menu item. Please initiate a synchronization. Once the error occurs, please select the File → Copy menu from the log window to copy the sync log to your clipboard for sending.

iOS

Login to Codebook for iOS and navigate to [More] > Sync. Long press on the Navigation Title “Sync” at the top, a new section will appear called “Debug”, within that section select the “Sync Log” row. The Sync Log will display, tap the “Copy to Clipboard” button and the Sync Log will be copied to the clipboard for sending.

And here are the instructions to reset your sync state with Google Drive using an overwrite and merge:

For anyone else looking on, please don’t perform these steps without explicit instructions from support as they are potentially destructive

1. Enable Sync Operations on the device which you believe to be most up-to-date by following these instructions: Codebook Help - Sync Operations
2. On that same device, set the Operation to Overwrite and start the Overwrite.
3. On every other device you’re syncing with, enable sync operations by following these steps: Codebook Help - Sync Operations
4. Set the operation to Merge and start the Merge. Choose Google Drive or Dropbox as the conflict authority.

Note: Choosing the conflict authority doesn’t choose all the data from one device, it’s only used in edge cases where the Standard conflict resolution can’t determine which record was updated more recently as described here: https://www.zetetic.net/codebook/documentation/sync-operations/#merge

[jhmpub]

Thank you for the clear instructions!

I did an overwrite / restore the hard way before I knew about debug mode and am not seeing any sync errors now.

If and when this issue occurs again, I will collect logs.