Decision Tree

class kerasy.ML.tree.DecisionTreeClassifier(criterion="gini", max_depth=None, random_state=None)

There are various simple, but widely used, models that work by partitioning the input space into cuboid regions, whose edges are aligned with the axes, and assigning a simple model to each region.

In this models, one specific model is responsible for making predictions at any given point in input space.

Within each region, there is an independent model to predict the target variable.

• Regression: simply predict a constant over each region
• Classification: assign each region to a specific class.

In order to learn such a model from a training set, we have to determine the structure of the tree about

• How many edges? (When to stop adding nodes?)
• How to chose the variable and threshold?

One simple approach is minimizing the objective function \(C(T)\)

• \(\lambda\) determines the trade-off.
• \(|T|\) is the number of leaf nodes.
• \(Q(T)\) is the loss functions like:
  • Residual Sum-of-Squares (Regression):
    $$Q_{\tau}(T) = \sum_{\mathbf{x}_n\in\mathcal{R}_{\tau}}\left\{t_n-y_{\tau}\right\}^2$$
  • Cross-Entropy (Classification):
    $$Q_{\tau}(T) = \sum_{k=1}^Kp_{\tau k}\ln p_{\tau k}$$
  • Gini index (Classification):
    $$Q_{\tau}(T) = \sum_{k=1}^Kp_{\tau k}\left(1 - p_{\tau k}\right)$$

※ \(p_{\tau k}\) is the proportion of data points in region \(\mathcal{R}_{\tau}\) assigned to class \(k\).