Metrics decide whether you're allowed to keep believing your own story. A startup without a disciplined measurement practice can talk itself into anything — "engagement feels strong," "users love it" — while the business quietly dies underneath the vibes. This guide covers the vocabulary and frameworks for turning raw activity into decisions: which numbers to trust, which to ignore, how to structure them into a system, and how to test whether a change actually worked. Use it to build the instrument panel you check before every roadmap call.
Vanity Metric vs. Actionable Metric
A vanity metric is a number that only goes up and feels good — cumulative sign-ups, page views, total downloads — but doesn't change any decision you'd make. An actionable metric changes your behavior by helping you pick a course of action.
This is the central distinction of Eric Ries's innovation accounting (from The Lean Startup): traditional accounting judges companies by revenue and growth, but a pre-product-market-fit startup has no reliable revenue to judge, so it needs a new accounting discipline built on metrics that can actually be acted on. Ries's own IMVU board decks looked exciting when they showed "total registered users" climbing — a classic "up and to the right" graph — but the number told nobody whether the product was getting better. It could only increase; it never went down, so it couldn't disconfirm anything. Croll and Yoskovitz (Lean Analytics) sharpen the test: if a metric changed, could you say with confidence whether that was good or bad, and would you know what to do about it? "Total signups" fails — it tells you nothing about what those users are doing. "Free-to-paid conversion rate this week" passes — if it drops, you have somewhere to look.
Worked example: two dashboards for the same app. Dashboard A shows "50,000 total downloads" (up every week, tells you nothing new). Dashboard B shows "18% of this week's new users completed onboarding, down from 24% last week" — a number you can and should act on today.
Watch out: vanity metrics aren't lies, and they aren't useless for a press release. The danger is using them to decide anything, or worse, presenting them at a board meeting as if they were evidence of progress — what Ries calls "success theater."
→ practice this in the cohort-thinking mission.
Innovation Accounting
Innovation accounting is Ries's three-step system for making startup progress falsifiable instead of hand-wavy: (1) use a minimum viable product to establish a real baseline of where the business stands today, (2) tune the engine — run experiments intended to move the baseline metrics toward the business plan's ideal, and (3) decide whether to pivot or persevere based on whether those experiments actually moved the needle.
The point is to replace "we shipped a lot this quarter" with "we learned this, and it moved metric X by Y." Ries's IMVU team spent months improving product quality, assuming that would fix flat sales — then discovered, once they tracked cohort-level conversion data instead of the misleading gross totals, that quality improvements weren't moving the metric that mattered at all. That's the mechanism working as intended: it forces you to admit when work isn't paying off, instead of shipping harder in the same direction.
Worked example: a team sets a baseline of 2% trial-to-paid conversion. They run three onboarding experiments over a month. Conversion moves to 2.1% — statistically indistinguishable from noise. Innovation accounting says: that engine tune didn't work, try a different lever, don't declare victory.
Watch out: innovation accounting only works if the baseline metric is actionable in the first place. Run it against a vanity metric and you'll "validate" changes that never mattered.
Retention and Churn
Retention is the share of customers still active (or still paying) after a given period — the clearest single signal that the product delivers ongoing value. Churn is its mirror image: the share of customers or revenue you lose in a period, and the leak that caps how big a subscription business can get. Monthly retention = 1 − monthly churn.
Churn is deceptively brutal because it compounds. A 4% monthly churn rate sounds harmless in isolation, but annualized (retention^12) it means you keep only about 61% of a cohort after a year — you've quietly lost almost 4 in 10 customers. OfficeDrop, a case study in Lean Analytics, made paid churn — paying users who downgrade or cancel, divided by paying users at the start of the month — its single most important number, because it doubled as a diagnostic: spikes in churn on a specific day pointed straight at a bug; churn among new customers pointed at a marketing/product mismatch; slow, steady churn among old customers pointed at feature direction drifting from what loyal users wanted.
Worked example: you start the month with 1,000 users and 40 cancel. Monthly churn = 40 ÷ 1,000 = 4%. Monthly retention = 96%. Compounded over 12 months, 0.96^12 ≈ 61% of the original cohort remains. Expected customer lifetime = 1 ÷ 0.04 = 25 months.
Watch out: the classic mistake is hiding churn behind growth — counting new signups in the denominator so the rate looks smaller. Always measure churn against the cohort that existed at the start of the period, and count only losses from that specific cohort.
→ practice this in the churn-and-retention mission, or run the numbers in the churn-retention calculator.
Conversion Rate and Funnel
Conversion rate is the percentage of people who complete a step you care about — visitor to sign-up, trial to paid — out of those who had the chance. A funnel is the ordered sequence of those steps from first touch to desired outcome, with a conversion rate at each stage showing exactly where prospects fall out.
Lean Analytics traces the funnel back to simple e-commerce flows (arrive, browse, add to cart, pay) but notes the modern version is a "Long Funnel" spanning search, social shares, and multiple return visits before a single conversion. Whatever the shape, the discipline is the same: measure the abandonment rate at each individual stage, not just the overall conversion, because that's where you find out which specific step is hemorrhaging prospects — sometimes something as narrow as one form field asking for information people don't want to give.
Funnels multiply, and multiplication is unforgiving: two stages that each convert at 5–10% turn 10,000 visitors into a few dozen customers. The end-to-end rate — not any single flattering stage — is the number that determines what you can afford to pay for a lead.
Worked example: 10,000 monthly visitors, 5% convert to leads (500 leads), 10% of leads convert to paying customers (50 customers) at $100 each = $5,000/month revenue. The overall conversion rate is 5% × 10% = 0.5% — meaning a paid click can cost at most about 50¢ before the funnel loses money on that customer's first month.
Watch out: quoting one flattering stage ("we convert 10% of leads!") while burying the end-to-end rate is the most common way funnels get spun to look healthier than they are.
→ practice this in the funnel-diagnosis mission, or model your own numbers in the conversion-funnel calculator.
North Star Metric and KPI / KPI Tree
A North Star Metric is the one metric that best captures the value customers get from the product — chosen so that moving it moves the business. A KPI (Key Performance Indicator) is any metric a team commits to watching and moving because it tracks something the business depends on; the North Star is simply the KPI everything else ultimately rolls up to. A KPI Tree is the diagram that decomposes that top metric into the sub-metrics ("Inputs") that mathematically or causally drive it, so the team can see which lever to pull.
The North Star Playbook (Amplitude) frames the North Star Framework as a metric plus a small set (3–5) of Inputs that collectively produce it — a "tree-like framework" of assumptions and causal relationships. Instacart's North Star, for example, was "total monthly items received on time," fed by four Inputs: number of customers placing orders, items per order, orders fulfilled, and on-time delivery rate. Each Input is something a specific team can actually move through their work — that's what separates a KPI tree from a vague company goal.
The book's checklist for a good North Star doubles as a checklist for good KPIs generally: it expresses customer value, reflects strategy, is a leading indicator (predicts future results rather than reporting past ones — which is why "monthly revenue" is a poor North Star, since it only tells you what already happened), is actionable, is understandable to non-technical people, is measurable, and is not a vanity metric. Daily Active Users, ad impressions, downloads, page views, and registered users all fail the checklist as North Stars — they say nothing about the value customers actually got.
Worked example: a delivery app rejected "scheduled deliveries" and "early deliveries" as North Star candidates because they didn't correlate with what customers valued. Research showed customers cared about deliveries that were simply on time and problem-free — "Happy Deliveries" — which did correlate with retention. That became the North Star, with Inputs like on-time rate and support-ticket rate feeding the tree beneath it.
Watch out: a KPI tree only works if the Inputs are things a team can actually influence day-to-day. A tree branch nobody owns is decoration, not a lever.
→ practice this in the north-star-metric and kpi-trees missions.
Cohort Analysis and Retention Curve
Cohort analysis means grouping users by when they arrived (or by a shared starting condition) and tracking each group over time, rather than blending everyone into one rolling average — so improvements and regressions aren't hidden by mixing old and new users together. A retention curve is the resulting plot: the share of a cohort still active over time. A curve that flattens means a core of users found lasting value; a curve that slides to zero means none did.
Lean Analytics makes the case with a revenue example: blended, average revenue-per-user looked flat and worrying month over month. Broken into cohorts, the picture flipped — the January cohort spent $5 in month one and tapered to $0.50 by month five, but the April cohort spent $8 in month one and $7 in month two. First-month spending was climbing fast and the drop-off pattern was improving too; a company that looked stalled was actually flourishing. Only cohort analysis revealed which number to focus on: drop-off after the first month.
The Cold Start Problem treats the shape of the retention curve as the core diagnostic for whether a product works at all: of users who install a typical app, 70% aren't active the next day, and by three months 96% are gone — the curve falls to zero. Networked products with strong engagement effects instead see the curve level out, meaning a durable core of users keeps coming back. A flattening retention curve is the classic quantitative signal of product-market fit.
Worked example: 500 users sign up at $30/month ARPU, retaining at 90% month over month. Month 1 revenue: 500 × $30 = $15,000. Month 2: $15,000 × 0.9 = $13,500. Month 3: $12,150. By month 6, only about 295 users remain (500 × 0.9⁵) and that month's revenue is roughly $8,857 (500 × $30 × 0.9⁵, since month-1 revenue is undiscounted). A naive projection of users × ARPU × months with no decay would claim $90,000 over six months; the real cohort generates about $70,284 — a 28% overstatement from ignoring decay.
Watch out: comparing a cohort's month-1 number to a different cohort's month-6 number and calling it a trend is a common cohort-analysis mistake — always compare cohorts at the same point in their own lifecycle.
→ practice this in the cohort-thinking mission, or run the decay math in the cohort-revenue calculator.
NPS
NPS (Net Promoter Score) is the percentage of customers rating their likelihood to recommend you 9–10 (promoters) minus the percentage rating 0–6 (detractors), typically on an 11-point scale collected from one survey question. Lean Analytics describes it as a practical proxy for virality in businesses — like enterprise sellers — where click-to-invite sharing isn't the norm: strongly enthusiastic customers are the ones who'll act as references, refer new business, or get quoted in marketing materials, so the metric captures word-of-mouth potential even when you can't track referral links directly.
Worked example: you survey 200 customers. 90 rate you 9–10 (45% promoters), 60 rate you 7–8 (neutral, ignored in the score), 50 rate you 0–6 (25% detractors). NPS = 45 − 25 = 20.
Watch out: NPS is a single noisy self-report, not a behavioral metric — treat a moving NPS as a prompt to go find out why (via the actual usage data), not as the final word on product health.
Engagement and DAU/MAU
Engagement is how often and how deeply users perform the product's core action — the behavior that sits between activation and long-term retention. DAU/MAU (daily active users ÷ monthly active users) is a stickiness ratio: what fraction of your monthly audience shows up on any given day.
Lean Analytics ties engagement directly to churn — "churn is the inverse of engagement" — and recommends segmenting engagement metrics by cohort so you can tell whether a product change actually made newer users behave differently, not just whether raw daily-active counts moved. A messaging or social product with a high DAU/MAU ratio (say, 50%+) has built a genuine daily habit; a B2B tool with a much lower ratio may still be healthy if its natural usage cadence is weekly, so the "right" DAU/MAU benchmark depends entirely on the product's expected use pattern — there's no universal target.
Worked example: a product has 3,000 monthly active users and 900 of them show up on a typical day. DAU/MAU = 900 ÷ 3,000 = 30% — meaning the average monthly user is active roughly 9 of 30 days.
Watch out: don't compare DAU/MAU across products with different natural cadences (a habit-forming game vs. a quarterly tax tool) — the ratio only means something against your own product's expected usage rhythm.
Leading Indicator
A leading indicator is an early metric that predicts a later outcome — like week-one activation predicting month-six retention — letting you act before the damage shows up in the lagging numbers. Lean Analytics contrasts this with lagging metrics, which report what already happened (quarterly bookings, this month's churn): useful as a baseline, but too late to change.
A good leading indicator has to be validated by actually checking the correlation, not just assumed. The book's example: customer complaint volume can be a leading indicator of churn — rising complaints predict future cancellations — but you only know that by testing whether complaint spikes historically preceded churn spikes in your own data, not by intuition. Activation is the canonical leading indicator in product analytics: the fraction of new users who reach a defined "aha" moment in their first session reliably predicts whether they'll still be around months later, well before you have enough elapsed time to measure actual long-term retention directly.
Worked example: you notice that users who complete 3+ actions in their first session retain at 45% after 90 days, versus 12% for users who complete fewer. First-session action count is now a usable leading indicator — you can optimize onboarding toward it today instead of waiting three months to see the retention number move.
Watch out: a leading indicator is only trustworthy once you've confirmed the correlation holds in your own data over at least one full cycle — borrowing someone else's leading indicator on faith is just a new vanity metric with better branding.
A/B Test
An A/B test is randomly splitting users between variants and comparing one metric, so the difference observed can be attributed to the change itself rather than to who happened to see which version. Lean Analytics calls this a "cross-sectional study" — different groups get different experiences at the same time — as opposed to cohort analysis, which follows the same group across time.
The book is candid about A/B testing's practical limit: unless you have Google- or Bing-scale traffic, you won't have enough visitors to run every test you want one factor at a time before your learning cycle stalls. Its Picatic case study shows the upside when it works — changing a single call-to-action from "Get started free" to "Try it out free" lifted click-through rate by 376% over ten days. When you have too many things to test and not enough traffic, multivariate analysis (testing several factors at once and using statistics to isolate which one correlates with the improvement) is the practical alternative.
Worked example: you run an A/B test on checkout button copy. Variant A ("Buy now") converts 400 of 5,000 visitors (8.0%). Variant B ("Get it today") converts 460 of 5,000 visitors (9.2%). The 1.2-point lift is the number you'd report — and only worth acting on once you've checked it's not just noise from the sample size.
Watch out: stopping a test the moment it looks like the variant you wanted is winning ("peeking") inflates false positives — decide your sample size and duration before you start, not after you like what you see.
→ practice A/B thinking alongside the funnel-diagnosis mission, where individual funnel steps are exactly what you'd test.
Go deeper
- Eric Ries, The Lean Startup — the definitive treatment of vanity vs. actionable metrics and innovation accounting; read it for the IMVU case study and the build-measure-learn loop this whole guide sits inside.
- Alistair Croll & Benjamin Yoskovitz, Lean Analytics — the deepest single source here: One Metric That Matters, cohort mechanics, leading/lagging indicators, A/B vs. multivariate testing, and stage-by-stage benchmark metrics by business model.
- John Cutler / Amplitude, The North Star Playbook — the North Star checklist, the Inputs tree, and real North Star case studies (Netflix, Instacart) for building a KPI tree that actually drives decisions.
- Andrew Chen, The Cold Start Problem — cohort retention curves as the core diagnostic for whether a network product is working, and how engagement effects reshape the curve as a network densifies.
- Wes Bush, Product-Led Growth — activation and time-to-value as the metrics that predict free-to-paid conversion in PLG businesses.