Subband adaptive filters have been proposed to speed up the convergence and to lower the computational complexity of time domain adaptive filters. However, subband processing causes signal degradations due to aliasing effects and amplitude distortions. This problem is unavoidable due to further filtering operations in subbands. In this paper, the problem of aliasing effect and amplitude distortion is studied. The prototype filter design problem is formulated as a multi-criteria optimization problem and all the Pareto optima are sought. Since the problem is highly nonlinear and nonsmooth, a new hybrid optimization method is proposed. Different prototype filters are used and their performances are compared. Moreover, the effect of the number of subbands, the oversampling factors and the length of prototype filter are also studied. We find that prototype filters designed via Kaiser or Dolph–Chebyshev window provide the best overall performance. Also, there is a critical oversampling factor beyond which the improvement in performance is not justified. Finally, if the length of the prototype filter increases with the number of subbands, an increase in the subband level will not deteriorate the performance.