Combining Predictors

Abstract

The most important theoretical tool in connection with machine learning is the bias/variance decomposition of error functions. Together with Tom Heskes, I have found the family of error functions with a natural bias/variance decomposition that has target independent variance. It is shown that no other group of error functions can be decomposed in the same way. An open problem in the machine learning community is thereby solved. The error functions are derived from the deviance measure on distributions in the one-parameter exponential family. It is therefore called the deviance error family.

A bias/variance decomposition can also be viewed as an ambiguity decomposition for an ensemble method. The family of error functions with a natural bias/variance decomposition that has target independent variance can therefore be of use in connection with ensemble methods.

The logarithmic opinion pool ensemble method has been developed together with Anders Krogh. It is based on the logarithmic opinion pool ambiguity decomposition using the Kullback-Leibler error function. It has been extended to the cross-validation logarithmic opinion pool ensemble method. The advantage of the cross-validation logarithmic opinion pool ensemble method is that it can use unlabeled data to estimate the generalization error, while it still uses the entire labeled example set for training.

The cross-validation logarithmic opinion pool ensemble method is easily reformulated for another error function, as long as the error function has an ambiguity decomposition with target independent ambiguity. It is therefore possible to use the cross-validation ensemble method on all error functions in the deviance error family.