I did something nasty just now. I rejected a PR merge that a lot of people put a lot of time and effort in and we universally seem to find a good idea. The github issue in question is:
We all want emojis and we’ve all wanted them for quite some time. Do a quick search on Slack and you get 336 results of conversations mentioning emojis. I bet most of them are around emojis as a signing phrase. @jarradhope even made a demo of it: https://rawgit.com/jarradh/ca8bfc407d7d19b8a5d69021fc442ef6/raw/48d9d2781709d959b7df9e8dea2807df8e5dddc9/emojiguard.html
Being one of the few products that actually has something in place that acts as a phishing guard is in itself quite unique and a showcase of how we value security.
While we have consensus that emojis are the right thing to do; we have not gotten the design right.
Here’s a rough table of issues we found in UX testing and how the latest implementation, to the best of my knowledge addresses them. https://docs.google.com/spreadsheets/d/1jAaPeoxVWUIy2zjdRWjCeRPPauUCZlSb3nb-NGJdHBM/edit#gid=0
In short, there are a few issues for which the emojis provide a solid solution and we are thereby introducing an improvement. The most important being:
The difference between the ‘seed phrase’ and ‘signing phrase’ was not clear to a number of participants.
Emoji signing phrase and Seedphrase are now conceptually and visually distinct.
Still not distinct enough for 100% of users, but let’s leave that for now.
There are about 6 issues that I’ve either seen remain in testing or imagine from a usability engineering perspective to expect to come up as issue sooner or later:
- The copy used does not indicate what one should do when not recognizing the emojis.
- None of the participants described intended behavior with regard to the phishing guard.
- No evidence about the effectiveness of the signingphrase
- The pattern does not prevent phishing for password.
- [Sign contract transaction] without introduction might create confusion for lay users. Nowhere in the app is there any mention of ‘Contracts’ yet.
- The critical moment does not show the Wallet address and amount for verification.
Although I find these issues highly important, they are not by definition a reason to reject a commit; Because all these issues also exist in the current implementation.
My main reasons to reject are process related:
- We should only release security-critical features if we have validated them thoroughly.
- After validation we should not make any changes to core UI elements without thorough analysis of the impact as changes my invalidate any findings we have about how secure our solution is.
- “It’s not going to make things worse” is not sound logic unless we can prove it. Honestly, this was my rationale up until this morning. Until I realized False security is worse than no security. I.e. a few improvements actually do make things worse.
- When we approve a commit knowing that issues remain we need to at least have a follow-up process defined (e.g. A github issue to resolve remaining issues)
- When we approve we need to make sure that the release is ‘contained’ and any issues are not unknowingly transferred to for example Desktop.
- The rationale that something is built is not a sustainable logic in a scaling organization. I could build something. It will be full of security holes, ugly and malfunctioning.
To kick of discussion, please share any criteria you’ve used in the past to judge when (not) to release new features.
Let me end this with some wise words from @Chad:
Security features need proper process or validation, otherwise it is going to end up on wall of shame 2.0