How to calculate information gain and entropy (decision tree example)

Entropy: how mixed is a node?

Entropy measures the uncertainty in a group. For two classes with shares p and 1 − p:

H = −p × log₂(p) − (1 − p) × log₂(1 − p)

• A pure node, where every loan defaulted or none did, has H = 0.
• A 50/50 node has H = 1 bit, the maximum for two classes.
• The root here has 8 defaults in 20 loans, p = 40%, so H = −0.4 × log₂(0.4) − 0.6 × log₂(0.6) = 0.971 bits.

In Excel: =-D2*LOG(D2,2)-(1-D2)*LOG(1-D2,2). LOG with base 2 gives the result in bits. Some libraries use the natural log, which changes the unit but never the ranking of splits.

Information gain step by step

Take the split "DTI above 40%". It sends 7 loans to the left (6 defaults) and 13 to the right (2 defaults).

• Entropy left: p = 6/7 = 85.7%, H = 0.592.
• Entropy right: p = 2/13 = 15.4%, H = 0.619.
• Weighted child entropy: 7/20 × 0.592 + 13/20 × 0.619 = 0.610.
• Information gain: 0.971 − 0.610 = 0.361 bits.

The weights matter. A tiny pure group must not count as much as a large mixed one, so each child is weighted by its share of the parent's rows. Without the weights, a split that isolates one loan would look perfect.

Worked example: three candidate splits on 20 loans

Rows 2 to 8 count the loans and defaults in the root and in the two children of each candidate split. Rows 11 to 13 combine each pair into weighted entropy, information gain and the Gini decrease. The lender and loans are fictional.

Worked example: three candidate splits on 20 loans

	A	B	C	D	E	F
1	Node	Loans	Defaults	p (default)	Entropy (bits)	Gini
2	All loans (root)	20	8	40.0%	0.971	0.480
3	Self-employed	8	5	62.5%	0.954	0.469
4	Salaried	12	3	25.0%	0.811	0.375
5	DTI above 40%	7	6	85.7%	0.592	0.245
6	DTI 40% or below	13	2	15.4%	0.619	0.260
7	Income under $40k	9	5	55.6%	0.991	0.494
8	Income $40k or more	11	3	27.3%	0.845	0.397
9
10	Split	Weighted entropy	Information gain	Weighted Gini	Gini decrease
11	Employment	0.869	0.102	0.413	0.068
12	DTI > 40%	0.610	0.361	0.255	0.225
13	Income < $40k	0.911	0.060	0.440	0.040
14
15	Best first split (highest information gain)		DTI > 40%
16	Split information of the DTI split		0.934
17	Gain ratio of the DTI split (C4.5)		0.387

Fictional loan data for practice only.

Comparing candidate splits

A tree tries every candidate question and keeps the one with the highest gain. That is how the classic ID3 and C4.5 algorithms grow a tree.

• Employment (self-employed vs salaried): gain 0.102.
• DTI above 40%: gain 0.361.
• Income under $40k: gain 0.060.

=INDEX(A11:A13,MATCH(MAX(C11:C13),C11:C13,0)) returns the winner. The tree then repeats the same search inside each child, on the rows that reached it, until the nodes are pure enough, too small or the maximum depth is reached. For numeric features such as DTI, every possible cut-point is a candidate split, and the best threshold is the one with the highest gain.

Counting the nodes with COUNTIFS

On raw data with one row per loan, COUNTIFS fills the node table: =COUNTIFS(DTI,">40%",Default,"Yes") counts the defaults among high-DTI loans and =COUNTIF(DTI,">40%") the loans. Divide to get p. A pure child has p = 0 or 1, and LOG(0,2) is an error in Excel, although 0 × log₂0 counts as 0 in the formula. Guard the entropy with =IF(OR(C2=0,C2=B2),0,…), as every entropy cell in the example does. The same counting pattern drives support, confidence and lift in market basket analysis.

Information gain vs Gini impurity

CART trees, and the default settings of scikit-learn, use Gini impurity instead of entropy: Gini = 1 − p² − (1 − p)². The root has a Gini of 0.480, and the DTI split lowers it to 0.255, a Gini decrease of 0.225. Gini is cheaper to compute because it needs no logarithm, and in practice both criteria choose the same split most of the time. They do here: DTI wins on both, ahead of employment (0.068) and income (0.040).

Gain ratio: the C4.5 correction

Information gain favours features with many values. A customer ID would split every loan into its own pure leaf and score the maximum gain while predicting nothing. C4.5 therefore divides the gain by the split information, the entropy of the split proportions themselves: for 7 and 13 loans it is 0.934, so the gain ratio of the DTI split is 0.361 ÷ 0.934 = 0.387. A split into many small branches has a high split information and is penalised.

To judge the finished tree, compare its predictions with the actual outcomes on held-out loans, see confusion matrix, precision and recall.

Common mistakes

• Not weighting the children. A simple average of the child entropies lets a tiny pure node dominate. Weight each child by its share of the rows.
• Dividing by all rows inside a node. p in a child is defaults in that child ÷ loans in that child, not ÷ all 20 loans.
• LOG of zero. A pure node makes LOG(0,2) an error. Return 0 for pure nodes with an IF.
• Mixing log bases. Bits (base 2) and nats (natural log) are both fine, but the parent and the children must use the same base.
• Trusting gain on ID-like features. A feature with a unique value per row has maximum gain and no predictive value. Use gain ratio or drop it.
• Choosing splits on the test data. Gains are computed on the training rows only. Score the finished tree on loans it has not seen.