customer_health_scoring v1.0.0

name: customer_health_scoring description: Build a customer health score people trust: pick usage and outcome signals that move before churn, normalize each to 0 to 100, weight them in the open, set bands from your own churned and retained accounts, add hard overrides for events a weighted average would hide, and validate the score against real renewals and churn before anyone acts on it. Use when someone asks to build, fix, audit, or explain a health score, choose health signals or weights, set red yellow green thresholds, or check whether a score actually predicts renewal.

Customer Health Scoring

Purpose

Build one number, 0 to 100, that a team can act on early enough to change the outcome. A health score earns trust three ways. Every signal in it moves before churn, not after. Every weight and threshold is written down with its reason, so a CSM can explain any score in one minute. And the score has been checked against real renewals and churn, with its misses named, before it drives a single play.

This skill covers the score itself: signal selection, normalization, weighting, bands, overrides, validation, and recalibration. It does not cover the full customer success motion around it. Pair it with a customer success playbook for onboarding, save plays, and expansion once the score is sound.

Most scores fail in predictable ways. They average in lagging signals that only confirm a decision the customer already made. They score missing data as zero, so a quiet healthy account looks critical. They use bands copied from a blog, fit to someone else's base rate. And nobody ever backtests them, so the team cannot tell a working score from a colored random number.

Workflow

1. Fix the outcome, the window, and the segment first

• Write down the outcome the score predicts, in plain words. Renewal, churn, contraction, or expansion are different outcomes and they do not share one best model.
• Fix the prediction window. A score meant to trigger action is usually read as risk in the next 90 days. Say the window on every report.
• Score each segment separately when renewal rhythm differs. Enterprise annual contracts, mid market quarterly terms, and self serve monthly plans move at different speeds. One set of bands across all three fits none of them.
• Record the decision the score drives. Name the play, the owner, and the deadline for each band before you pick signals. If no action follows from a band, the band is decoration.

2. Inventory candidate signals, then classify them

• List every signal you could get reliably, with its source system and refresh cadence. Product analytics, billing, support desk, CRM activity, and survey tools are the usual sources.
• Classify each candidate two ways. Leading or lagging: does it move weeks before churn, or only confirm it after. Outcome or activity: does it show the customer got value, or only that someone was busy.
• Prefer leading outcome signals. Core feature use, milestone and outcome attainment, seat or license activation, and sponsor engagement move early and tie to value. Cancellation requests, non renewal notices, and final payment failures are lagging. Keep lagging signals for validation and overrides, not as weighted inputs.
• Check coverage before you fall in love with a signal. A survey score that exists for 20 percent of accounts cannot carry 20 percent of the weight for the other 80 percent.

Full catalog with definitions and failure modes: references/signal-catalog.md.

3. Run the selection tests and keep four to six signals

Keep a signal only if it passes all five tests:

1. Timing test: it changed before past churns, in your own data, not just in theory. Look at churned accounts 30, 60, and 90 days before they left.
2. Separation test: retained and churned accounts score visibly differently on it. If the two distributions overlap almost fully, it adds noise.
3. Action test: a low value points to a concrete play. If the team cannot name what they would do differently, cut it.
4. Coverage test: it is present and fresh for most accounts in the segment. Sparse signals get small weight or become flags, not weighted inputs.
5. Honesty test: the team cannot game it easily, and the customer cannot inflate it by accident. Raw login counts fail here when a single integration or admin login inflates them. Use value actions, not visits.

• Start with four to six signals. More than eight usually means several signals measure the same thing and double count it.
• Cut vanity signals on sight: total registered users, total data stored with no link to value, page views, emails sent. They look busy and predict little.

4. Normalize every signal to 0 to 100

• Give each signal a written scoring rule before any weighting. Use bands, percentiles within the segment, or a capped ratio. Pick one method per signal and document it.
• Bands work best for a first model because anyone can audit them. Example for login recency in a daily use product: active in the last 3 days scores 100, 4 to 7 days scores 70, 8 to 14 days scores 40, 15 to 30 days scores 15, over 30 days scores 0.
• Normalize counts by size. Tickets per active user, active users as a share of licensed seats, usage per account baseline. Raw counts punish large healthy accounts and flatter small inactive ones.
• Score trend as its own input when the level alone misleads. Usage at 60 percent of the account's own trailing average is a different fact from usage steady at that level for a year. Rolling 7 day and 30 day averages smooth one day spikes.
• Handle missing data by redistributing weight, never by scoring zero. If a signal is NA for an account, spread its weight across the present signals in proportion to their weights. Gainsight scorecards do exactly this for NA measures. A missing survey response means unknown, not unhappy.
• Age stale inputs down. A satisfaction response from a year ago is weaker evidence than one from last month. State the freshness window per signal and drop inputs older than it into the missing rule above.

5. Weight in the open

• Assign weights that sum to 100 percent. Publish the weights, the reason for each, and the date they were set, in the same place the score is shown.
• Start from a hypothesis, label it as one, then fit it to outcomes in step 8. A common starting shape for B2B software is usage and adoption 35 percent, outcome attainment 20 percent, support health 15 percent, engagement and relationship 20 percent, commercial hygiene 10 percent. That shape is a starting point from published models, not a standard. Your data gets the final say.
• Group related signals before weighting when they measure one thing. Weight the group once, then split inside the group. This stops three usage metrics from quietly becoming 70 percent of the score.
• Keep manual scores small and rubric bound. A CSM pulse can carry weight, but only with a written 1 to 5 rubric, for example 5 means quarterly reviews held with an economic buyer present, 3 means contact with managers only, 1 means no meeting can be scheduled. Unrubriced opinion at high weight destroys trust in the whole score.
• Record every weight change like a code change: who, when, why, and what validation result justified it.

Worked models for three business shapes: references/worked-models.md.

6. Add hard overrides for events a weighted average hides

A weighted average lets strong usage bury a fatal event. Fix that with overrides, applied after the weighted sum:

• Cap the score when a hard event is present. Examples: champion or economic buyer left, active cancellation or downgrade request, failed payment unresolved, open severity 1 issue past its service target near renewal, legal or procurement block stated in writing.
• Set the cap per event and write it down. A common pattern is a cap at 39, the top of the critical band, while the event is open. Clear the cap when the event resolves, and log both changes.
• Use exceptions the other way too. High ticket volume with fast resolution and good satisfaction should not drag a thriving account into red on volume alone. Gainsight scorecards support this kind of exception rule for the same reason.
• Overrides explain themselves in the output. Show the weighted score, the override that fired, and the final score. Never silently replace the number.

7. Set bands from your own distributions

• Do not copy bands. Published bands range from critical below 30 to critical below 60 on the same 0 to 100 scale, because base rates differ. Fit yours to your accounts.
• Plot or tabulate the scores of accounts that churned and accounts that renewed in the last two to four cycles. Set the critical line where it catches a useful share of churners without flagging so many accounts that the team cannot work them. A red band that fires on 40 percent of the book is not operational.
• Give each band four fields: score range, what it means, the play it triggers, and the response deadline. Example shape: Healthy, standard cadence and expansion check. At risk, owner outreach this week and a written root cause. Critical, manager review within 48 hours and a save plan with a named executive sponsor.
• Always show score plus trend plus top driver. A 72 falling fast and a 72 rising are different accounts. The driver line names the one or two signals that moved the score most, so the owner knows where to start.

8. Validate against real outcomes before launch

Do not skip this step. A score that has never been backtested is an opinion with decimals.

1. Rebuild historical scores at a fixed point before the outcome, usually 60 days before renewal or churn. Scoring with data from after the decision leaks the answer into the test.
2. Build the confusion matrix at your critical threshold. Predicted at risk means score below the threshold. Actual positive means the account churned or did not renew.
3. Report sensitivity, specificity, accuracy, and precision, with the counts behind them. False negatives, healthy on paper and churned anyway, cost more than false positives in most books. Read the two error types separately and say which one your threshold favors.
4. Check band separation. Report the actual churn rate inside each band. A working score shows a steep step: critical churns far more than at risk, which churns far more than healthy. Flat bands mean the score does not separate, whatever its accuracy headline says.
5. Beat a simple baseline or do not ship. Compare against one strong signal alone, such as seat utilization or core feature use. If the composite does not beat the baseline on the same historical window, ship the baseline and keep working.
6. Sweep the threshold. Run the bundled validator at several thresholds and pick the point where the cost balance fits your team capacity. More flagged accounts than owners can work is a threshold problem, not an effort problem.
7. Check each segment separately. A model that works on mid market and fails on enterprise needs segment models, not a global tweak.

Full protocol, formulas, and a worked backtest: references/validation.md. Run it with bin/health_score.py validate.

9. Test weight changes one at a time

• Change one weight or one signal at a time, rerun the same backtest, and keep the change only if separation or the error balance improves on historical data.
• Run a sensitivity check before arguing about a weight. Move it by 10 points each way in the backtest. If the result barely moves, stop debating it and spend the time on signal quality.
• Lock the model during a measurement window. If weights change weekly, no validation result means anything, because the thing being measured keeps changing.

10. Operate, explain, and recalibrate

• Recalculate on a fixed cadence and keep history. Weekly fits most subscription businesses. Daily overreacts to normal gaps, monthly reacts too late to save a renewal. Whatever you choose, store dated snapshots so trend and backtests stay possible.
• Show the formula wherever the score appears. Trust comes from a CSM being able to click any score and see signals, normalized values, weights, overrides, and last refresh time.
• Let owners challenge a score, with a receipt. A manual adjustment needs a reason, a name, and an expiry date. Track override accuracy in the next validation round. A score nobody may question gets ignored, and a score anyone may quietly edit gets gamed.
• Recalibrate quarterly, and after any pricing, packaging, product, or segment change that alters what usage means. Compare predicted bands to actual outcomes, review the largest false negatives first, and publish what changed and why.
• Watch the score itself for drift: share of accounts per band, average score by segment, and flag volume versus team capacity. A sudden shift after a data pipeline change is a data bug until proven otherwise.

Output Contract

Deliver these items, in this order:

1. Model card: outcome predicted, prediction window, segment, refresh cadence, owner, model version, and the date weights were last validated.
2. Signal table: each signal with definition, source system, scoring rule, weight, freshness window, and the play a low value triggers.
3. Scoring sheet or run output: per account, the normalized signal values, weighted contributions, weighted score, any override fired, final score, band, trend, and top driver.
4. Band definitions: ranges with meaning, play, owner, and response deadline for each band.
5. Validation report: historical window, snapshot point, sample size by segment, confusion matrix with counts, sensitivity, specificity, accuracy, precision, churn rate by band, baseline comparison, and the threshold chosen with its reason.
6. Change log: every weight, signal, band, or override change with date, author, reason, and the validation result behind it.
7. Open limits: signals wanted but not available, segments with too few outcomes to validate, and known blind spots.

Run the scorer with bin/health_score.py score <account.json> and the validator with bin/health_score.py validate <history.csv>. Both print JSON plus a short human summary.

Operating Rules

• Never score missing data as zero. Redistribute its weight and mark the signal missing.
• Never present a band, weight, or accuracy figure copied from another company as if it fits this book. Starting shapes are hypotheses. Validation on your own outcomes makes them yours.
• Never act on an unvalidated score at scale. Pilot it, backtest it, read its misses, then roll it out.
• Never hide the formula. If a score cannot be explained to the account owner in one minute, simplify it until it can.
• Never let a weighted average bury a hard event. Champion loss, cancellation intent, and unresolved payment failure cap the score.
• Never mix outcome definitions or snapshot timing inside one validation run. One outcome, one window, one snapshot point.
• Never report accuracy without the counts and the base rate. 90 percent accuracy on a book with 5 percent churn can mean the score caught nobody.
• Always pair the score with trend and top driver. A bare number sends owners hunting for the reason the score exists to give them.
• Always compare against a one signal baseline in validation. Complexity that does not beat simplicity does not ship.
• Keep the model small. Four to six signals, each passing the five selection tests, beat twelve signals nobody can explain.
• Recalibration is part of the job, not a rescue. Quarterly at minimum, with the change log updated in the same pass.

Supporting files

• references/signal-catalog.md: signal domains, candidate signals with definitions, sources, direction, normalization hints, and the failure mode to watch for each.
• references/validation.md: backtest protocol, confusion matrix formulas, band separation checks, threshold sweep method, segment checks, and a worked backtest with real arithmetic.
• references/worked-models.md: three complete models, seat based software, usage based platform, and high touch services, each with weights, scoring rules, overrides, and a scored example account.
• bin/health_score.py: scorer with missing data redistribution, trend delta, and hard override caps, plus a validator that builds the confusion matrix, band churn rates, and a threshold sweep from a history file.