Low-Power Lossy Networks (LLNs) are a type of Internet of Things (IoT) meshnetwork that collaboratively interact and perform various tasks autonomously. TheRouting Protocol for Low-power and Lossy Network (RPL) is the most used rout-ing protocol for LLNs. Recently, we have been witnessing a tremendous increasein attacks on Internet infrastructures using IoT devices as a botnet (IoT botnet).This thesis focuses on two parts: designing an ML-based IDS for 6LoWPAN, andgenerating a new larger labeled RPL attack dataset by implementing various non-attack and attack IoT network scenarios in the Cooja simulator. The collected rawdata from simulations is preprocessed and labeled to train the Machine Learningmodel for Intrusion Detection System (IDS). We used Deep Neural Network (DNN),Random Forest Classifier (RFC), and Support Vector Machines with Radial-BasisFunction kernel (SVM-RBF) learning algorithms to detect attack in RPL based IoTmesh networks. We achieved a high accuracy (96.7%) and precision (95.7%) usingthe RFC model. The thesis also reviewed the possible placement strategy of IDSfrom cloud to edge.