Customer churn: who is about to leave, and why
My first tree memorized the data and did worse than guessing. Pruning it fixed that, and showed which tenure and billing patterns signal a customer is about to leave.
AUC on held-out customers
test accuracy, above the 73.8% baseline
customers, 26.6% of whom left
levels deep, small enough to explain
Why churn
Every business with repeat customers has the same problem: you usually find out someone’s leaving after they’ve left. I wanted a model that flags them while there’s still time to do something, and tells you what to look at.
The data, and the trap in it
7,032 telecom customers and 19 attributes: tenure, contract, services, billing and payment method. 1,869 of them (26.6%) left. I converted the categories to numbers and split the data 80/20.
Here’s the trap: 73.8% of the test customers stayed. So a model that just says “everyone stays” is 73.8% accurate and completely useless. Accuracy alone was never going to tell me much, so I judged the models on the ROC curve and AUC.
The tree that memorized everything
| Model | Train accuracy | Test accuracy | Test AUC |
|---|---|---|---|
| Guess that everyone stays | 0.738 | ||
| Unpruned decision tree | 0.998 | 0.733 | not reported |
| Pruned decision tree (max depth 4) | 0.791 | 0.785 | 0.815 |
My first decision tree scored 99.8% on the training data. It looked great. Then it met new customers and dropped to 73.3%, worse than guessing that everyone stays. It had memorized the training set instead of learning anything.
The fix was simple and effective: I limited the tree to four levels. Training accuracy came down to 79.1%, test accuracy went up to 78.5%, and the two landed close together, which is exactly what you want. AUC came in at 0.815.
The unpruned tree’s mistakes show why this matters for a real team. Of 369 customers who left, it caught 186, and it flagged 192 loyal customers by mistake. That’s a retention team chasing the wrong people about half the time.
What it flagged
Four attributes carry most of the decisions
Tenure, monthly charges and total charges travel together. They’re really one story: how long someone has been around and how much they’ve paid. Fiber-optic service is the one that stands on its own, and if I saw that at a real company, I’d go looking for a price or service problem.
How I’d use it
Anywhere with recurring customers or clients: subscriptions, software, property management, client relationships in a services firm. Swap the features and the target, keep the workflow. A ranked list sends outreach to the people where it will actually change something.
And because the tree is only four levels deep, I can hand the rules to the teams who’d act on them. Nobody has to trust a black box.
Before it goes live
- Real data. This was public telecom data. A real version needs the company’s own customer data and privacy controls.
- Better models. A single tree is where you start. Ensembles are the next step, which is what I did in the credit risk case study.
- Proof. A score doesn’t prove outreach works. You need a defined action and a holdout group to measure the difference.
- Drift. Customers change, so the model needs monitoring and a trigger to retrain.