In this paper, a real-time unmanned aerial vehicles (UAVs) detection framework is proposed for GPU embedded applications. To achieve this, this paper proposed a new modified model based on You Only Look Once (YOLO) to detect multi-UAV in aerial images with a complex background. The proposed CNN architecture uses five inception module and dilated convolution with two different factors to increase detection accuracy of YOLOv3 tiny, while preserving its computational time. To further improve the detection accuracy, Generalized Intersection over Union loss is replaced with the bounding box regression loss in the original YOLOv3 tiny loss function. To obtain higher frame rate, scalable kernel correlation filter (sKCF) algorithm is integrated to the detection model. More specifically, the proposed UAV detection method is used to initialize the sKCF tracker at every nth frame. Thus, the detected UAVs can be detected in intermediate frames with a low memory footprint. The proposed model and compared models are trained and tested on a variety of training and test airborne videos, captured under different outdoor scenarios. The average precision of the proposed model is 0.8265 and achieves 32 FPS performance on Jetson-TX2. The results show that the proposed model which is widened with inception module using dilated convolutions has a very high performance in terms of accuracy and speed even when it is compared with deep CNN architectures such as Darknet 53. © 2021, The Optical Society of India.