How to clean up after import

[rickcogley]

I imported an export of a fairly big 1password database, to try to give Codebook a real test.

First question is about field labels. The field labels used by the export are apparently whatever is used by the webforms, from which 1password captures its data. Therefore, there’s a real myriad of field names or labels, like Password, pwd, passwd, pass1, and so on, just for password fields.

• How can I clean these up? I suppose one choice is to wipe the db and clean up the field names, but merging those in Excel sounds like a torturous process.
• How can I find data that has a specific field?
• If I convert all data that has a field like “pwd” to Password, if the field “pwd” becomes blank will it disappear? If not, how can I delete it from the list - File, Customize labels?

Second question is about Notes. When I imported, I got a bunch of notes in a category “Secure Note” and, they all have their notes stored in a notesPlain field. Is there a better way to import these so they go directly into Notes, with a proper Notes field? Those look a lot nicer.

Third question is on Icons. There appears to be no way to bulk set icons. Am I wrong? Now I’ve got many, many entries with a brown folder icon. How can I set it in bulk?

Also, fourth, a general observation: the import failed several times. I used the conversion tool, but, the complex data that had been exported into the 1pif file seemed to confuse it, because the exported csv simply did not work to import at all. Then I used Calc to clean up a bit, deleting any fields that were all blank. There were many of these, and, I found a bunch of oddly formatted fields, that somehow were not escaped and were munging things up, so I deleted those.

When I got the csv cleaned up, finally Codebook appeared to import, but it would crash partway through. The progress got further each time, and finally succeeded.

Now however, the performance of the app is pretty sluggish. For instance, when I am in looking at the Labels list, converting fields to Password and clicking the mask checkbox has a significant lag. Also when performing standard ops in the UI, like revealing the password, it skips a couple beats before revealing.

My Macbook pro is relatively new, purchased a few months ago, so I don’t think it’s that the machine is underpowered.

Please advise, thanks.

[developernotes]

Hello @rickcogley

Thank you for your candid feedback regarding Codebook, I’ve tried to address your questions below inline:

@rickcogley

The field labels used by the export are apparently whatever is used by the webforms, from which 1password captures its data. Therefore, there’s a real myriad of field names or labels, like Password, pwd, passwd, pass1, and so on, just for password fields.

How can I clean these up?

Unfortunately this would require manual modification/clean up of the output CSV file from the conversion tool before importing directly within Codebook. It would be difficult to automate this process without also having a manual process inline to understand what labels you wish to rename fields to.

@rickcogley

How can I find data that has a specific field?

Codebook allows you to search on the content of your data, unfortunately we do not expose a mechanism for you to search for a given field name for example. This is something we might consider for the future, but currently we do not have plans for at the moment.

@rickcogley

If I convert all data that has a field like “pwd” to Password, if the field “pwd” becomes blank will it disappear? If not, how can I delete it from the list - File, Customize labels?

I’m not sure what you mean by the field “pwd” becoming blank, can you explain that a bit more? Customizing the label text has no impact on the stored values associated to it. If you change the text value of the label within the label editor, that change will propagate to all labels that were previously using that name (i.e., pwd → Password). If you delete that label within the label editor, if will ask you if you wish to delete its usage across all entries that contain it, as those fields would be removed from their respective entries.

@rickcogley

Second question is about Notes. When I imported, I got a bunch of notes in a category “Secure Note” and, they all have their notes stored in a notesPlain field. Is there a better way to import these so they go directly into Notes, with a proper Notes field?

With our latest release of Codebook for Windows and Codebook for OS X, we made an adjustment to allow import of entries as our new note entry, as opposed to a standard entry that contains a note field. Unfortunately the two desktop versions are currently using a different value to distinguish a note entry on import. If you wish to import a specific entry as a new note in Codebook for OS X, please use the column name Entry Note, if you wish to import a specific entry as a note in Codebook for Windows please use the column name Note Entry. We will consolidate this column name to be the same across platform in a future release, my apologizes for the difference.

@rickcogley

There appears to be no way to bulk set icons. Am I wrong? Now I’ve got many, many entries with a brown folder icon. How can I set it in bulk?

We do not have a mechanism for batch editing icons, as it is not something often requested. Would you select a series of entries and change them from all of one icon to all of another icon, are you trying to do this with categories, or do you have something else in mind?

@rickcogley

Now however, the performance of the app is pretty sluggish. For instance, when I am in looking at the Labels list, converting fields to Password and clicking the mask checkbox has a significant lag. Also when performing standard ops in the UI, like revealing the password, it skips a couple beats before revealing.

Would you mind sharing the size of the data set you are working with, we would like to investigate this further. Specifically under the File → Get Info… menu item, could you provide the database size, category, entry, field counts, as well as the replica CSN value?

[rickcogley]

Hi Nick - please see inline -

Rick Cogley
Mobile: 090-9959-5452 (Outside Japan, remove “0”. +81-90-…)

[developernotes]

Hello @rickcogley

Would you mind reposting your response when you get the opportunity, it doesn’t look like your previous message made it through.

[rickcogley]

I emailed it back, writing inline, but that did not go over well. I’ll paste it in:

Thank you for your candid feedback regarding Codebook, I’ve tried to address your questions below inline:

JRC - my pleasure, and I appreciate that, thanks.

rickcogley:
The field labels used by the export are apparently whatever is used by the webforms, from which 1password captures its data. Therefore, there’s a real myriad of field names or labels, like Password, pwd, passwd, pass1, and so on, just for password fields.

How can I clean these up?

Unfortunately this would require manual modification/clean up of the output CSV file from the conversion tool before importing directly within Codebook. It would be difficult to automate this process without also having a manual process inline to understand what labels you wish to rename fields to.

JRC - it’s probably simply better to re-do it. I have an unbelievable number of fields from that import.

rickcogley:
How can I find data that has a specific field?

Codebook allows you to search on the content of your data, unfortunately we do not expose a mechanism for you to search for a given field name for example. This is something we might consider for the future, but currently we do not have plans for at the moment.

JRC - noted.

rickcogley:
If I convert all data that has a field like “pwd” to Password, if the field “pwd” becomes blank will it disappear? If not, how can I delete it from the list - File, Customize labels?

I’m not sure what you mean by the field “pwd” becoming blank, can you explain that a bit more? Customizing the label text has no impact on the stored values associated to it. If you change the text value of the label within the label editor, that change will propagate to all labels that were previously using that name (i.e., pwd → Password). If you delete that label within the label editor, if will ask you if you wish to delete its usage across all entries that contain it, as those fields would be removed from their respective entries.

JRC - certainly. What I want to say is, if I convert all data which was labelled as “pwd” to the proper “Password” field type, will “pwd” then disappear? It’s a bit moot at this point, because I think re-doing the import is best.

rickcogley:
Second question is about Notes. When I imported, I got a bunch of notes in a category “Secure Note” and, they all have their notes stored in a notesPlain field. Is there a better way to import these so they go directly into Notes, with a proper Notes field?

With our latest release of Codebook for Windows and Codebook for OS X, we made an adjustment to allow import of entries as our new note entry, as opposed to a standard entry that contains a note field. If you wish to import a specific entry as a new note, please use the column name Note Entry.

JRC - ok, that I can do.

rickcogley:
There appears to be no way to bulk set icons. Am I wrong? Now I’ve got many, many entries with a brown folder icon. How can I set it in bulk?

We do not have a mechanism for batch editing icons, as it is not something often requested. Would you select a series of entries and change them from all of one icon to all of another icon, are you trying to do this with categories, or do you have something else in mind?

JRC - yes, some kind of batch edit mechanism might be useful for people to organize their things. If you could multi-select, then do, say, Edit, Batch Edit, and then make a change that makes sense in that context, like changing the icon, or, changing a category (if even such a thing exists in Codebook), that would be helpful. Generally speaking.

rickcogley:
Now however, the performance of the app is pretty sluggish. For instance, when I am in looking at the Labels list, converting fields to Password and clicking the mask checkbox has a significant lag. Also when performing standard ops in the UI, like revealing the password, it skips a couple beats before revealing.

Would you mind sharing the size of the data set you are working with, we would like to investigate this further. Specifically under the File → Get Info… menu item, could you provide the database size, category, entry, field counts, as well as the replica CSN value?

JRC - please see https://drive.google.com/open?id=0B3E_g6R5WF8AY1c1NFdDMVhfYkk

[developernotes]

@rickcogley

JRC - certainly. What I want to say is, if I convert all data which was labelled as “pwd” to the proper “Password” field type, will “pwd” then disappear? It’s a bit moot at this point, because I think re-doing the import is best.

Changing the name within the label editor from “pwd” → “Password” will just rename the field, thus any entry that contained a field with the name of “pwd” would now display “Password”.

@rickcogley

JRC - yes, some kind of batch edit mechanism might be useful for people to organize their things. If you could multi-select, then do, say, Edit, Batch Edit, and then make a change that makes sense in that context, like changing the icon, or, changing a category (if even such a thing exists in Codebook), that would be helpful. Generally speaking.

That is an interesting idea, we will discuss this internally, however note we do not have any immediate plans for this.

Thank you for sharing the database statistics information, that will be helpful as we look into performance behaviors with different database sizes.

[rickcogley]

Another thing I noticed is, the converter is extremely sluggish. The buttons you click to specify the files take minutes to respond. No idea what that’s about. None of my other software is doing that.

[rickcogley]

By the way, after a reset of the db, and import of a very cleaned up csv, Codebook is a lot snappier. Same number of records, so, it seems like having that crazy number of fields I had, was the culprit.

[developernotes]

Hi @rickcogley

Thanks for getting back to us, we are glad to hear cleaning up the fields improved the responsiveness.