Data Hostage Crisis: ColorNote & Google

This post comes in two parts – the fix, and the drama. Figure, if you are here for the fix, let’s disclose that up front (unlike how the rest of the internet works). So here it is:

• If you use ColorNote and all your data has vanished (multiple reasons how this can happen), I have found things you can try, but some of it is a bit technical.
• Here is the the Github project that worked for me and the README there also contains other options that are easier (but didn’t do the trick for my situation)
• Note: making it work with Oracle’s JRE is not fun, but the easy button there is to use Azul Zulu’s OpenJDK.
• For the record, after installing the JDK I confirmed Java by running: java -version

• The above project makes a precompiled jar available (if you are not technical, this will be weird, but just run the command from wherever you downloaded the jar file – it’s worth a shot!); which I downloaded and used:

java -jar colornote-decrypt.jar 0000 28 < {path to your backup file} > notes.json

• I will hopefully do a follow-up post, because the resulting “notes.json” file is not easy to work with, but for now, I am just thrilled to see my data. It lives! Later I will try to clean it up and post more about that.

Alright, like most people, I pour my life into a cadre of applications for a variety of reasons. This tale abstractly harkens back to Vint Cerf’s lecture on Digital Vellum, which I got to enjoy sitting 3 seats from the podium in a small, but visually stunning, library room at Georgetown University. In this particular case, I have poured a bunch of content into a free application called ColorNote. Now, to ColorNote’s credit, they have a support address and I have not been patient enough to await a response, and they have a cloud sync, which I was using for years! But the sync failed me, and what unfolded was a chess game. Disaster recovery shouldn’t be a chess game – it should be tic-tac-toe with first move privileges.

Also, because this article is about to shred ColorNote, I think it is probably fair to inject a caveat – “free” software doesn’t pay well unless it gets picked up by a well-sponsored community. I’m not mad at the people trying to do something with ColorNote, I think this is likely symptomatic of new team members picking up a previous torch, and cutting corners to make ends meet while continuing to deliver their “free” software.

Where Things Failed

I think in larger patterns, hence the reference to Digital Vellum, but essentially, as the software evolved, the decisions made by the software team were not thoroughly contemplated. Meaning, there are likely more people out there who will experience this issue in the future for a variety of potential reasons. Maybe this blog post will help some of those folks!

As ColorNote has evolved, they have changed their backup format. Again, to their credit, whether it is SQLite, Doc files or .backup files, they were at least making backups of customer data on the customer’s local device. This, albeit messy and space consuming, was the mechanism that allowed me to “restore” my data.

Here is where the axe got raised in my scenario: When I started using ColorNote, I had a gmail account. Then, several evolutions of the product later, I stopped using that gmail account, and had it deleted. I then started using a new gmail account. ColorNote was never aware of the switch and didn’t seem to care. I had my backups being synced to that old account for probably two years or more without ColorNote telling me those backups were failing. That’s a significant product design flaw.

And, of course, this story wouldn’t be complete without that raised axe being dropped. Here’s how that happened: I installed ColorNote on a new tablet and there was no data to sync from my Google account. I didn’t know why, so I opened the version on my cell phone, where all my data was still present and after poking around, I saw the issue. My tablet was syncing to my current gmail account, which had no data. My cell phone was syncing to my old gmail account (well, it wasn’t really syncing because that account was deleted). I thought, “I have all my data on my cell phone, why don’t I update my sync to my new account and then all the content on my phone should show up on my table.” Upon changing the sync settings (log out of old account, login to new account), instead of taking all my data and syncing it to the new location… it deleted everything on my cell phone. That is also a significant design flaw.

I thought, well this is bad.

Into the Product Design Abyss

Gather up your headlamp, a trustworthy climbing rope and some snacks because we are going to descend into the depths of this debacle from a product design perspective.

• First – I tried the backups. The product was maintaining several years of backups on my cell phone. These were, luckily, still on the phone. Bad design, but bad design in my favor. Thank you. But alas, selecting a previous backup and attempting to restore it said I needed to login to my old gmail account. More bad design, this really should have been thought through a whole lot better!
  
• Next – I hooked a USB cable to my phone, found the backups and copied them to my local drive. The files were binary or compressed. I began looking at the internet for away to open these files. That’s when I learned of all the different backup formats (.db, .doc, .backup); however everyone was talking about the master password. I didn’t use the master password setting. In my case, somewhere across their version history, they decided that files should “belong” to a Google account, and didn’t verify the account was even still legitimate before forcing my data to be “protected” by requiring authentication to the account. Naughty design, to bed without supper for you!
  
• Then – I had bright idea. Resurrect my Google account. About an hour of grinding on that idea and one sad support chat later, it became clear that the account, once closed could not be restored (the error is “account could not be found”). And it couldn’t be recreated because, they report “the account is still in use.” When I took that to support, they said, for security reasons, Google never releases an account back into the pool for reuse. That is, actually, a fair-ish answer given their scale.
  
• Finally, I visited the open source community on GitHub, searching for projects that matched “colornote”. I am clearly not alone. There are pages of color note projects. Some of them are not related, but there are several projects that transfer ColorNote to other platforms or decrypt the backup files. Ultimately I picked one of those projects to get at my data, and it worked.

Break out a snack as we dangle in the darkness. The descent continues, because even though I have my data, it’s not usable in its current format. It’s just not deleted. I will have to write an update to this post or a subsequent post to truly put a bow on this topic, but there’s plenty to still consider beforehand:

• There are multiple issues with how the ColorNote product handles customer data
  
• No Import/Export: The product has no mechanisms to import or export data. And to clarify, I mean importing data from an open format, or exporting data to an open format (JSON, CSV, even XML would be more open than their current file structure). ColorNotes only contain textual data, so this is not that hard to implement. Without this capability the product blocks customers from truly owning their data.
  
• Faulty Cloud Sync: The product isn’t letting customers sync to their own cloud. It syncs to who-knows-where-in-ColorNote-land. It is basically just syncing a proprietary file to a proprietary endpoint. Again, customers do not own their data.
  
• Faulty Protection: The files cannot be used unless the person logs into the account that owns the file. But look at how that gets established – how does ColorNote “know” a file belongs to someone. Well, in this case, they write that into the file. The reason people have successfully been able to “decrypt” these files is because they just skip over the part of the file that declares which account or declares the master password protecting it. The rest of the data is there, available. So, this protection is enough for busy muggles, but it doesn’t provide ANY protection against someone who knows enough of what they are doing.
  
• Faulty Backup Mechanics: This saved me. Once I got the files off my phone and on to my computer I could see that the primary backup was 1K (essentially empty) and all the previous backups were over a megabyte. But at some triggering event, the software creates a complete backup (blockchain style, lol). There are dozens of these files, each with ALL the data plus whatever is new. So we got local copies, synced copies, and a rumored SQLite database of the same data if folks are using a rooted phone. Customer data is getting spammed into two locations.
  
• Faulty Sync Mechanics: Touched on this before, but the application started tying backups to a user account without even verifying the account was still valid. Then, when a user attempts to update the software with a new account, it not only deletes all that data, it leaves copies of the previous account data on the user’s device. So there is no scenario where this is a good idea.

Thud. Wow, here we are at the bottom of the descent. We’ve got a pile of unusable data, an application that was so poorly designed it both blocked me from getting to my data by trying to erase it all AND allowed me to save my data via one of its spammed backups AND blocks me from importing that after I clean it up because it has no import feature.

Ultimately, I don’t trust this application anymore, so I will have to move the data elsewhere. And this could lead into a diatribe on how when software is “free” then we are not the primary customer or driver for its evolution. It also is a great case study of how an app can be fantastic until something goes wrong and then we found out who our real friends are, but for now, I am going to sit at the bottom of the design abyss and finish my snacks. Hey, is that a velvet Elvis painting? Thankyuverymuch.

Image Credits

Today’s image came from our AI friends at Midjourney

Prompt: google and colornote tried to lock me out of my account but I found a way, thanks to open source!
Result: https://www.midjourney.com/jobs/ce97e8ff-8fe2-4198-85f8-0d2f29d0b1e8?index=0

Prompt: a team of diverse and well-equipped spelunkers wearing headlamps are descending via ropes into a deep cave.
Result: https://www.midjourney.com/jobs/a1c74123-b310-44b2-9677-9dffc7c2ad5f?index=0

Prompt: Elvis Presley’s face, oil-painted on a black velvet canvas, in an ornate picture frame, sitting on the floor of a cave
Result: https://www.midjourney.com/jobs/a1c74123-b310-44b2-9677-9dffc7c2ad5f?index=0