Image sync failing?

Codebook
1

Hello - I have an entry on Codebook 4.5.7 (1078) on my iPhone, with some photos I took of documents on my iPhone camera, that just won’t sync to my Codebook 4.5.8 (1033) on Mac.

I’ve tried vacuuming and checking integrity on both sides. In either case, it said:

1 Recovered Fields were moved to the Recovered Data entry within the Recovered Category.
3 Recovered Images were moved to the Recovered Data entry within the Recovered Category.

… for the integrity check, but, repeating it on iPhone yields a blank (no error), while the error just recurs on the Mac.

When I look at the Recovered folder, I see one entry on both the iOS and Mac sides, with no images to speak of, despite their mention in the above.

One thing I was thinking it might be is, the images could be saved as HEIC. Would that have any effect? Any ideas how I can troubleshoot?

Kind regards,
Rick

2

Right, and, here’s how I managed to fix it.

  1. On the iPhone side, selected each of the images in the troublemaking entry and exported to my Mac, noting in Finder that they are all jpeg, not HEIC.
  2. Synced both sides, but no change.
  3. Ran db integrity and vacuum on both sides, but no change.
  4. Edited the iPhone side entry, removed the four backed up images. Saved the entry.
  5. Sync, integrity, vacuum again on both sides, but no change.
  6. Deleted the entry on iPhone, synced again. Once that entry was gone, now Vacuum does not give any error on Mac.
  7. Created a new entry with slightly different name on Mac. Synced, and confirmed it exists on iPhone.
  8. Ran the images through “image optim” on Mac, and reduced their size from quite large, 7 Mb each to about 1 Mb each. Edited entry and added the images.
  9. Synced, and now the images are indeed available on both sides.

Not sure what would have caused this but, it appears something got corrupted. Now it’s ok.

3

Hi @rickcogley

Thank you for your longtime support of Codebook and for reaching out, although I’m sorry to hear about the trouble with images added to your iPhone not syncing to your Mac.

Codebook for iOS and macOS supports importing and displaying HEIC images (Codebook for Windows/Android won’t display HEIC’s but they will still sync). If they are above the the maximum image size threshold, you should have been prompted to compress them (which would convert them to JPEG format), but from your description it sounds like they were below that threshold (10MB): Codebook Help - Image Attachments

From my interpretation of the description of the issue you experienced, it sounds like one of a couple of things could have gone wrong:

Something Went Awry With Your Sync State

Each Codebook client tracks what changes have been seen from other clients/remotes it syncs with. This allows it to only pull down/apply changes from the other client/remote which haven’t already been seen by this client and only push up/send changes which haven’t been seen by the other client/remote. If something had gone awry with this tracking, a client could have thought it already pushed a change to the remote which it didn’t or a another client could have thought it already applied a change which it didn’t.

Something Went Awry With the Change Capture or Upload (on the iOS device)

Codebook captures and stores changes for syncing as edits are made in the interface. Then when syncing with a remote it determines which changes it has locally which haven’t been seen by this other client/remote and uploads them. If an error occurs when attempting to capture a change or upload it, we attempt to properly handle it within Codebook, but if somehow this error wasn’t gracefully handled (or a hard crash occurred), it could potentially cause an issue with the change not properly being synced to other devices.

I noticed that you did write in to support separately with a crash report from one of your iOS devices when attempting to sync with Google Drive, so this could theoretically fall under the second description there.

Because this issue occurred for you, to make sure all your Codebook data is up-to-date and properly synced with all your devices I’d like to have you perform a one time operation to reset your sync state. This will re-establish your sync tracking/remote with the Codebook data present on all your Codebook devices. These initial Merges will take longer than a standard sync as its comparing/replaying all your Codebook data from each device involved.

Before performing these operations, if you don’t use Time Machine or some other mechanism to have an additional backup of your encrypted Codebook database, I’d like to have you take a manual backup of your current Codebook data as an added precaution:

Take a Manual Backup

  1. Sync all your Codebook devices so that your Mac has the most recent Codebook data present.
  2. Quit and close Codebook on your Mac.
  3. Open Finder and Go menu > Go to Folder and paste this location in the address bar: ~/Library/Group Containers/PD7G6HRMGV.net.zetetic.STRIP/Strip
  4. Within that location is a default.strip package, this contains all your current encrypted Codebook data. Copy that package and paste a copy onto the Desktop.

Here are the steps to perform the one time overwrite/merge to reset your sync state

If you’re syncing with Google Drive or Dropbox

  1. Enable Sync Operations on the device which you believe to be most up-to-date by following these instructions: Codebook Help - Sync Operations
  2. On that same device, set the Operation to Overwrite and start the Overwrite.
  3. On every other device you’re syncing with, enable sync operations by following these steps: Codebook Help - Sync Operations
  4. Set the operation to Merge and start the Merge. Choose Google Drive or Dropbox as the conflict authority.

If you’re syncing over WiFi

  1. Enable Sync Operations on your mobile device by following these instructions: Codebook Help - Sync Operations
  2. Set the operation to Merge on your mobile device and start the Merge. Choose whichever device you feel to be “most correct” as the conflict authority.
  3. Repeat steps 1 & 2 for every other mobile device you’re syncing with.
  4. If you have any other Desktops/Laptops, repeat steps 1 & 2 with one of the mobiles and the other Desktops/Laptops.

Note: Choosing the conflict authority doesn’t choose all the data from one device, it’s only used in edge cases where the Standard conflict resolution can’t determine which record was updated more recently as described here: Codebook Help - Sync Operations

After performing these steps, make a test change and perform a standard sync operation to confirm that it syncs properly.

If this issue does occur again for you after resetting your sync state, we’d definitely like to dig further into it to determine what could have caused it. To do this, we’d request that you leave your devices in the state that they’re in after the sync doesn’t propagate and we’d have you collect the sync logs from each device and then provide you with a tool to (locally) inspect your changesets (chunks of changes) in the remote (or device if syncing over WiFi) which should help diagnose where the issue occurred.

Please let me know if you’re able to reset your sync state using those instructions and if there’s anything else I can do to assist. Thanks!

Cheers,
Micah

4

Hi Micah, thanks for the details.

Well noted about HEIC, thanks.

Sync appears to be working again, so, should I still reset the sync state?

5

@rickcogley

If sync is working again and you’re relatively certain all your Codebook data is correct/matches on all devices, then you don’t necessarily need to reset your sync state.

That being said, it doesn’t necessarily hurt anything to do so and there are a couple of benefits to doing it at this point (since we know at one point a change didn’t sync for you):

Consolidate Change History

Resetting your sync state consolidates all your change history in the remote into a couple of changes representative of all your “current” Codebook data. As you sync your Codebook data over time the codebook changeset files (chunks of encrypted changes) build up over time. If for example, you’ve added some data + synced, a week later modified that same data + synced, and then another week later deleted that same data + synced, there would be 3 changesets (1 representing the insert, 1 the modification, and another for the delete). Another example would be you add data + sync, a week later modify that same data + sync, and then another week later modify that data again + sync, there would again be 3 changesets. Maintaining this changeset history is necessary when we’re not sure which clients have seen which changesets.

When doing the overwrite Codebook deletes this entire change history in the remote and uploads a changeset representative of your “current” data state (on the device performing the overwrite). When doing a Merge Codebook will combine the data on the remote with the local data (ignoring any previous deletes) and only upload the difference of what’s local which hasn’t been seen by this remote. There’s a higher level overview of what each of the advanced sync operations do on this page: Codebook Help - Sync Operations

Ensure All Your Data Is Matching

Because this issue occurred for you and we can’t say for certain which of the issues in my previous post it could have been caused by, this would ensure that any other data which may have not previous synced properly (if it was a “sync state” issue rather than a “change capture or upload” issue) would be properly synced.

Baseline For If The Problem Occurs Again

If this issue was caused by a “sync state” issue rather than a “change capture or upload” issue, if it were to occur again, it would be harder to determine whether it was caused by the previous “sync state” issue (since we never reset the sync state) or whether it was a new “sync state” issue. I do suspect that the issue you experienced here was indeed related to “change capture or upload” issue though since everything seems to be syncing fine for now.

Just to clarify for anyone else reading this who hasn’t had an issue, resetting your sync state shouldn’t be necessary (although could be used if you’re running low on space on Google Drive/Dropbox and would like to consolidate your encrypted change history).

1 Like
6

ok, makes sense, I’ll give it a try then

7

Took me long enough but, this cleanup / merge let me switch to desktop wifi sync. Thanks!

8

@rickcogley

Absolutely. Glad I could help.

Please feel free to reach out again if you have any further questions, issues, or feedback.

Happy Holidays!