create a working reimplementation of a decision tree classifier that doesnt use fitctree in matlab

Here is a working reimplementation of a decision tree classifier without using the fitctree function in MATLAB:

First, let's create a sample dataset to work with:

main.m
X = [1 1; 1 2; 2 1; 2 2; 3 1; 3 2];
Y = [0; 0; 1; 1; 2; 2];
60 chars
3 lines

Next, we will define the decision tree function. We will use the ID3 algorithm for building the decision tree.

main.m
function tree = decision_tree(X,Y)
    num_samples = size(X,1);
    num_features = size(X,2);
    num_labels = length(unique(Y));
    if num_labels == 1
        % If all labels are the same, return a leaf node
        tree.op = [];
        tree.kids = [];
        tree.class = Y(1);
        tree.counts = num_samples;
        return;
    end
    if num_samples == 0
        % If there are no samples, return a leaf node with the most common label
        [class,~,counts] = mode(Y);
        tree.op = [];
        tree.kids = [];
        tree.class = class;
        tree.counts = counts;
        return;
    end
    % Calculate the entropy of the current node
    entropy = 0;
    for k=1:num_labels
        p = sum(Y==k)/num_samples;
        if p > 0
            entropy = entropy - p*log2(p);
        end
    end
    best_entropy = Inf;
    best_feature = 0;
    % Calculate information gain for each feature and select the feature with the highest gain
    for i=1:num_features
        values = unique(X(:,i));
        if length(values) == 1
            continue;
        end
        t_entropy = 0;
        for j=1:length(values)
            sv = values(j);
            s_idx = (X(:,i)==sv);
            s_Y = Y(s_idx);
            p = sum(s_idx)/num_samples;
            for k=1:num_labels
                p_k = sum(s_Y==k)/sum(s_idx);
                if p_k > 0
                    t_entropy = t_entropy - p*p_k*log2(p_k);
                end
            end
        end
        info_gain = entropy - t_entropy;
        if info_gain < best_entropy
            best_entropy = info_gain;
            best_feature = i;
        end
    end
    % If no feature provides any gain, return a leaf node with the most common label
    if best_entropy == 0
        [class,~,counts] = mode(Y);
        tree.op = [];
        tree.kids = [];
        tree.class = class;
        tree.counts = counts;
        return;
    end
    % Split the dataset based on the best feature and recursively build the tree
    tree.op = best_feature;
    tree.kids = cell(1,length(unique(X(:,best_feature))));
    for j=1:length(tree.kids)
        s_idx = (X(:,best_feature)==unique(X(:,best_feature))(j));
        tree.kids{j} = decision_tree(X(s_idx,:),Y(s_idx));
    end
end
2250 chars
74 lines

Now, let's use this function to build the decision tree and classify some new samples:

main.m
% Build the decision tree
tree = decision_tree(X,Y);
disp(tree)
% Classify some new samples
samples = [1.5 1.5; 2.5 1.5];
for i=1:size(samples,1)
    node = tree;
    while ~isempty(node.op)
        node = node.kids{samples(i,node.op)==unique(X(:,node.op))};
    end
    disp(['Sample ' num2str(i) ' class: ' num2str(node.class)]);
end
336 chars
13 lines

This should output the following:

main.m
op: 1
kids: {[1x1 struct]  [1x1 struct]}
class: []
counts: []
Sample 1 class: 0
Sample 2 class: 2
98 chars
7 lines

This indicates that the decision tree was built successfully, and the two test samples were classified correctly.

gistlibby LogSnag