We will call an estimator for the regression function defined by the CART methodology a regression tree. The word CART means classification and regression tree. This chapter will focus only on the regression trees.
Regression trees are regression function estimators that are constant in rectangels. The rectangles need not have equal size, as in the case of the (standard) histogram estimators. Regression trees have the special property that they are representable as binary trees. This makes the graphical presentation of estimates possible, even in the case of many regression variables. Indeed, the regression tree is especially useful in the multidimensional cases. Furthermore, the regression tree has the advantage that it works also when the regression variables are a mixture of categorical and continuous variables. The response variable is assumed to be a continuous variable. Regression tree is well suited for the random design as well as for the fixed design. The theory of regression trees was developed by Breiman et al. (1984).
CART methodology consists of three parts.
First, we grow a regression tree which overfits the data.
Secondly we prune from the overfitting tree a sequence of subtrees
and lastly we try to select from the sequence of subtrees a subtree which
estimates the true regression function as best as possible.