A permutation problem arises in the case of locating multiple speech sources using several sensor arrays in the far field. The intersection of different direction of arrival (DOA) estimates between sensor arrays leads to a set of real source locations as well as a set of false intersections. This paper presents a novel method for pairing DOA estimates from different sensor arrays, resulting in the corresponding real intersection points. The algorithm presented is numerically efficient and suitable for real time implementations. Real room recordings are used to evaluate the method.