Context. Cloud computing (CC) is developed as a Human-centered computing model to facilitate its users to access resources anywhere on the globe. The resources can be shared among any cloud user which mainly questions the security in cloud computing. There are Denial of Service and Distributed Denial of Service attacks which are generated by the attackers to challenge the security of CC. The Next-Generation Intrusion Prevention Systems (sometimes referred as Non-Traditional Intrusion Prevention Systems (NGIPS) are being used as a measure to protect users against these attacks. This research is concerned with the NGIPS techniques that are implemented in the cloud computing environment and their evaluation.

Objectives. In this study, the main objective is to investigate the existing techniques of the NGIPS that can be deployed in the cloud environment and to provide an empirical comparison of source mode and destination mode in Snort IPS technique based on the metrics used for evaluation of the IPS systems.

Methods. In this study, a systematic literature review is used to identify the existing NGIPS techniques. The library databases used to search the literature are Inspec, IEEE Xplore, ACM Digital Library, Wiley, Scopus and Google scholar. The articles are selected based on an inclusion and exclusion criteria. The experiment is selected as a research method for the empirical comparison of Source mode and destination mode of Snort NGIPS found through literature review. The testbed is designed and implemented with the Snort filter techniques deployed in the virtual machine.

Results. Snort is one of the mostly used NGIPS against DoS and DDoS attacks in the cloud environment. Some common metrics used for evaluating the NGIPS techniques are CPU load, Memory usage, bandwidth availability, throughput, true positive rate, false positive rate, true negative rate, false negative rate, and accuracy. From the experiment, it was found that Destination mode performs better than source mode in Snort. When compared with the CPU load, Bandwidth, Latency, Memory Utilization and rate of packet loss metrics.

Conclusions. It was concluded that many NGIPS of the cloud computing model are related to each other and use similar techniques to prevent the DoS and DDoS attacks. The author also concludes that using of source based and destination based intrusion detection modes in Snort has some difference in the performance measures.