Thanks everyone so far for all your thoughtful comments, ideas and contributions! There are a lot of good ideas here for sure, and it is great to see so many people outside of Core having ideas for how to make our main product work better and retain more users. Special kudos to Barry for getting us sweet actionable retention rate graphs!
Weekly update
As of July 1, per post by Jakub earlier, we no longer count end to end tests in metrics. However, this didn’t have as a big an impact as one might suspect.
Unfortunately the dashboard at Grafana seems to be down. However, we can still the latest in Kibana.
DAU: Sunday to Sunday - 2658 on Jul 5, 2985 on Jun 28. This is -11% negative growth. Since this is on Sunday, the e2e test impact is negligible.
Retention rate: No new data here, see post by Barry above for latest here. However, since no releases or specific targeted marketing has occured in the last week (to my knowledge), there’s little reason to believe things would’ve changed here.
Android install numbers
A good of checking accuracy of metrics is to cross-check it with different sources. Here are the Android installs per week. These are exported from Play Store and you can see them in the Status Growth Metrics spreadsheet.
How to action on retention
This question was brought up earlier here. We obviously won’t be able to get accurate segmentation or anything of the sorts. But we can make do.
What the retention rates in Barry’s post above allow us to do is to have hypotheses about cohorts and campaigns. For example: this feature will increase D2-D30 retention. Then we can check the cohort for when that version is released.
We can also use it to test campaigns. Say there’s a big spike in installs a specific week. Then we can check how users from this cohort are retained. If they retain worse than previous cohorts, perhaps they aren’t our target audience.
In the future, we should be able to distinguish Android, iOS and desktop peers here. Though I’d leave that kind of sophistication as out of scope for now and more of a bonus until we get basics in place.
A good marketing/onboarding campaign here would be: how can we get N users to use Status for a week or a month? In the Saas world, this is a very common strategy for onboarding and increasing retention.
What’s good?
Since last week, a lot of things have improved when it comes to the state of metrics.
- We have clear DAU and retention metrics that tell us where we succeed and fail.
- We have a dashboard that clearly shows DAU and stickiness for everyone.
- We have increased confidence in our metrics by removing end to end testing from metrics collection.
- We have a very clear understanding of how bad the situation is right now.
- We have retention rate graphs that can be used for experiments and analysis on a time scale of ~days.
Outside of metrics and visibility, we have also seen a lot of good ideas on how to improve our growth and retention in this thread. And, of course, we have seen a lot of product development happening that we believe will result in better retention, e.g. images, push notifications, and other initiatives.
What still needs improvement?
I’ll divide this into two parts, one is general and the other is specific.
General:
- The hard part: growth and retention is still terrible. This week went backwards.
- Mindset: I am not convinced we are doing our best here. See section below.
- Theory of retention: I still don’t see it, and I’m not convinced by our reasoning here. This needs clear thinking where we know exactly who our users are and why they keep using Status, or not.
Specifics:
- Methodology, mentioned in OP, still needs work. This needs to be crystal clear and requires Core involvement to be correct. See https://notes.status.im/metrics-method?both# for a start. Marketing is an example user here.
- Retention rate dashboard and/or script. With Barry’s notebook we have actionable retention metrics. It’d be useful if we could script this, so a CSV export is run and we can update these graphs on a daily basis.
- Solve mystery 1. In mid-April we had a spike of installs through a performance marketing campaign. Question: Why does this not translate into a spike in DAU?
- Solve mystery 2. We see DAU of around ~3k, but installs per week recently is around ~100. Question: How does that add up?
- Different types of metrics. It’d be useful to see other, wallet/dapp-browser-oriented metrics reported in this thread.
Mindset
A word of warning - this section might be controversial and you might not agree with it. I have been having this conversation with a few people privately over the last week, and I think it is important to say publicly for everyone.
For us to do this, we need to have the right mindset. This means keeping growth and retention front and center every day. And I’m largely talking about Core here, as that’s the product team with the biggest impact.
It is also important that there’s a feeling of a sense of agency from the product team in this. Growth isn’t some magic sauce that another team solves. What (performance) marketing does is adding fuel to the fire. But there has to be a fire to begin with.
If we don’t currently feel this sense of agency or control, then that’s a bug to fix.
I heard in the Core retro last week that we shouldn’t let ourselves get distracted by metrics. I said it then but I’ll say it again - we should let the metrics guide us, and constantly ask ourselves if what we are making will improve retention or not. We have the data now, so there are no excuses not to do this.
I also heard that we shouldn’t let discussions take away from building. That seems dangerous to me. We are still a startup, and we don’t know exactly what the right path forward is. We need to be open and flexible. The more reflection we can do here, as shown in this thread, the better. To me this is where something like a theory of retention comes in, because it allows us to think more deeply than just “let’s get to feature parity with every other messenger out there”. This is a slight strawman position, but as if that’s the reason people would switch to Status. It doesn’t personally resonate with me as a user anyway.
There are a lot of smart and thoughtful individuals in this organization, and it’d be a shame not to capitalize on that. Six hours to chop down a tree and spend first four sharpening the ax kind of thing.
Obviously this should be complemented by ruthless execution once we are confident in our choices. But it shouldn’t stop us from breathing and making sure we are on the right path. Judged on a weekly growth basis, we are not, and haven’t been for the last three months.
It might seem like we have to do a lot of things at the same time. I don’t think that’s the case. As a core product, we mostly have to do one thing: improve retention. Usually only very few actions have a big impact. And that starts by taking a deep breath and emptying the cup in your mind. Take a long walk and a big breath.
Over to you!
No one has ever done what Status is doing, thus no one can predict how the product will be used once it’s ready for the masses. New users can (almost) always be acquired through paid advertising, but retention requires a killer feature set. Push notifications may feel like a silver bullet, but a solid use case for Status is also critical… and that means clear messaging on how its components stack up against other apps, and how its core features are better together.
friends!
