calculate information gain using matlab

An interactive tool to understand how Information Gain is calculated, a core concept in machine learning algorithms like decision trees, often implemented in environments like MATLAB.

---

---

Data Split Summary

Node	Total Samples	Positive Class	Negative Class	Entropy (bits)

What is to calculate information gain using matlab?

Information Gain is a core concept in machine learning and data mining used to determine the effectiveness of an attribute in classifying a dataset. It is the fundamental criterion used by decision tree algorithms, such as ID3 and C4.5, to select the optimal feature for splitting the data at each node of the tree. While software like MATLAB provides powerful toolboxes (e.g., the Statistics and Machine Learning Toolbox™) that automate processes like decision tree construction, understanding how to calculate information gain using MATLAB concepts is crucial for any data scientist. The calculation quantifies the reduction in uncertainty (entropy) about the target variable after the dataset is split based on the values of a particular attribute.

The goal is to find the split that results in the highest information gain, as this leads to the “purest” possible child nodes—that is, nodes where the samples predominantly belong to a single class. A higher information gain means the chosen attribute is more informative for making a classification decision. This principle is not just theoretical; it’s the engine behind how tools like MATLAB’s fitctree function decide on their splitting rules automatically.

The Information Gain Formula and Explanation

Information gain is calculated by subtracting the weighted average of the children’s entropies from the parent’s entropy. Entropy itself is a measure of impurity or disorder in a set of examples.

Formula

The general formula for Information Gain is:

Gain(S, A) = Entropy(S) - ∑ [ (|Sv| / |S|) * Entropy(Sv) ]

The formula for Entropy (for a two-class problem) is:

Entropy(S) = -p+ * log2(p+) - p- * log2(p-)

Formula Variables

Variable	Meaning	Unit	Typical Range
Gain(S, A)	The Information Gain from splitting dataset S on attribute A.	bits	0 to 1 (for a binary target)
Entropy(S)	The impurity of the entire dataset S before the split.	bits	0 (pure) to 1 (maximum impurity)
Sv	The subset of S for which attribute A has value v.	–	–
|Sv| / |S|	The weight of a child node; the proportion of samples in the child node.	Unitless ratio	0 to 1
p+	The proportion of positive class examples in a set.	Unitless ratio	0 to 1
p-	The proportion of negative class examples in a set.	Unitless ratio	0 to 1

For more details on decision trees, check out this guide on understanding decision trees.

Practical Examples

Example 1: High Information Gain

Imagine a starting dataset (parent node) with 100 samples, perfectly balanced with 50 positive and 50 negative instances. This set has maximum entropy (1.0 bit). We find a feature that splits it into two “pure” child nodes:

• Input: Parent (100 total, 50 positive), Child 1 (50 total, 50 positive), Child 2 (50 total, 0 positive).
• Logic: Parent entropy is 1.0. Child 1 entropy is 0. Child 2 entropy is 0. The weighted child entropy is (50/100)*0 + (50/100)*0 = 0.
• Result: Information Gain = 1.0 – 0 = 1.0 bit. This is a perfect split, providing maximum information.

Example 2: Low Information Gain

Now consider the same parent node (100 samples, 50 positive). This time, the split is not as effective:

• Input: Parent (100 total, 50 positive), Child 1 (50 total, 30 positive), Child 2 (50 total, 20 positive).
• Logic: Parent entropy is 1.0. Child 1 entropy is ~0.971. Child 2 entropy is ~0.971. The weighted child entropy is (50/100)*0.971 + (50/100)*0.971 = 0.971.
• Result: Information Gain = 1.0 – 0.971 = 0.029 bits. This split provides very little information and is not much better than the original, un-split data. A better feature should be chosen.

How to Use This Information Gain Calculator

This calculator simplifies the process of understanding how to calculate information gain using MATLAB concepts by focusing on a single binary split. Follow these steps:

1. Enter Parent Node Data: Input the total number of samples and the number of samples in the positive class for your initial dataset.
2. Enter Child Node 1 Data: After an imaginary split, input the total samples and positive class samples that fall into the first group.
3. Enter Child Node 2 Data: Do the same for the second group. The tool assumes a binary split, so the sum of child samples should ideally match the parent total.
4. Interpret the Results: The calculator instantly updates the entropies and the final information gain in “bits”. A higher value indicates a more effective split.
5. Analyze the Chart and Table: Use the visual chart to see the reduction in entropy and the summary table to review the numbers for each node.

A deeper dive into practical implementation can be found in our MATLAB Machine Learning Toolbox guide.

Key Factors That Affect Information Gain

• Parent Node Purity: If the parent node is already very pure (low entropy), the maximum possible information gain from any split will be low.
• Child Node Purity: The goal is to achieve child nodes that are as pure as possible (entropy close to 0). The purer the children, the higher the gain.
• Split Balance: The size of the child nodes matters. A split that sends all but one sample into one child node is often less useful, even if that one sample is perfectly classified.
• Number of Classes: The calculation becomes more complex with more than two target classes, but the principle of reducing entropy remains the same.
• Choice of Attribute: This is the most critical factor. A good attribute will naturally separate the classes, while a poor one will result in mixed, high-entropy children.
• Handling Continuous Data: For numerical features, a threshold (e.g., “Age < 30") must be found to create a binary split. Information gain is calculated for many possible thresholds to find the best one. For more, see data preprocessing techniques.

FAQ

1. What is a “good” information gain value?

It’s relative. The “best” split is the one with the highest information gain among all possible splits at that node. A value closer to the parent’s entropy is better, with a maximum of 1.0 for a perfectly balanced binary problem.

2. What is the difference between Information Gain and Gini Impurity?

Both are metrics for measuring impurity in a node. Gini Impurity is often faster to compute as it doesn’t involve a logarithm. In practice, they usually result in very similar trees. Explore our comparison article on Entropy vs. Gini Impurity for more.

3. Can information gain be negative?

In standard implementations, no. A split is only made if it reduces entropy. In theory, a poorly designed split could lead to a negative value, but algorithms are designed to avoid this.

4. How is this actually used in MATLAB?

Functions like `fitctree` in the Statistics and Machine Learning Toolbox use information gain (or Gini impurity) as the ‘SplitCriterion’ to automatically build a decision tree from your data, selecting the best features and split points.

5. What does an entropy of 0 or 1 mean?

An entropy of 0 means a node is perfectly pure; all samples belong to the same class. An entropy of 1 (in a binary classification) means the node is perfectly impure, with a 50/50 split of classes.

6. Why use log base 2?

Using log base 2 results in a unit of “bits” for entropy, which has roots in information theory. It represents the number of yes/no questions needed to convey the information. Other bases like the natural log (ln) can be used, which would result in units of “nats”.

7. What if I have more than two child nodes (e.g., for a categorical feature with 3 values)?

The formula extends naturally. You calculate the entropy for each of the three child nodes and then compute the weighted average of all three before subtracting from the parent entropy.

8. How do I implement this myself?

Start with our tutorial on implementing the C4.5 algorithm, which uses Information Gain Ratio.