It is argued that in applications of concept learning from examples where not every possible category of the domain is present in the training set (i.e., most real world applications), classification performance can be improved by integrating suitable discriminative and characteristic classification schemes. The suggested approach is to first discriminate between the categories present in the training set and then characterize each of these categories against all possible categories. To show the viability of this approach, a number of different discriminators and characterizers are integrated and tested. In particular, a novel characterization method that makes use of the information about the statistical distribution of feature values that can be extracted from the training examples is used. The experimental results strongly supports the thesis of the paper.