Olga Berezovsky

You need to know each event in your analytics: what it means, when it’s fired, and when it’s not fired. Only then can you create your KPIs and create your reporting.

It’s very easy to misread retention in tools like Amplitude if you rely on “any user activity”. You might be over-reporting activities (e.g., due to notifications triggering an activity) and therefore retention. Plus, retention might not be proportional for different segments. You can also miss events happening outside of your app (e.g., change of plans directly in App Store settings).

There are two different types of A/B tests. The process and KPIs you’re looking at will change based on each.

1. New feature launch: more market research, connect more with your baseline metrics, run forecasts, look at how the population of users might move, and how it will affect the premium offering.
2. Improvement/Optimization of an existing feature: a more straightforward approach. Formulate a hypothesis, make sure it’s even worth testing, estimate the test runtime for significance, ensure you have the right analytics events, launch to 5%, and if everything is good, ramp up, and run analysis at the end.

Develop your A/B testing process and framework within the company before looking for tools.

The main metrics you should track can depend on your growth stage. Three that generally give a good view of your subscription business:

• Install to trial

• Trial to paid

• Churn rate

It’s very challenging to draw the line between winbacks that happen organically and winbacks that occur because of specific efforts. Example: people subscribe before summer, cancel in September, and resubscribe after Christmas.

When looking at retention, it matters how you define “active users”. The way you define it should fit into any other engagement metrics you typically track (is it session start, app opens, screen views, main user action, etc.?).

Pros and cons for in-house vs. buy for A/B testing tools:

• In-house: struggle with the flexibility it allows

• Tools: struggle with integrating with analytics

Create two different retention metrics: one for free users and one for paid users. For subscriptions, it’s usually easier to look at churn instead of retention.

It’s tough to move retention. Improvements are typically not seen within months; it can take years.

Two main types of retention:

• X-day retention: on which day a % of users come back (e.g., D1, D7, D30)
• Unbounded retention: you define a range (e.g., Days 1-5, or Day 5-7)

For subscription businesses, it’s recommended to use X-day retention because you can tie things to your free trial period.

Notifications are a good way to get users to re-subscribe. Segmenting and targeting notifications to specific groups often requires matured analytics.

Pricing/offering iterations are very impactful and crucial for the business, but also the most difficult types of tests. Any price testing starts with marketing research.