Malware has been one of the key concerns for Information Technology security researchers for decades. Every year, anti-malware companies release alarming statistics suggesting a continuous increase in the number and types of malware.  This is mainly due to the constant development of new and more sophisticated malicious functionalities, propagation vectors, and infection tactics for malware. To combat this ever-evolving threat, anti-malware companies analyze thousands of malicious samples on a daily basis, either manually or through semi-automated means, to identify their type (whether it's a variant or zero-day) and family. After the analysis, signature databases or rule databases of anti-malware products are updated in order to detect known malware.  However, due to the ever-growing capabilities of malware, the malware analysis process is challenging and requires significant human effort. As a result, researchers are focusing on data-driven approaches based on machine learning to develop intelligent malware detectors with high accuracy. Specifically, they are focused on extracting static features from malware in the form of n-grams for experimental purposes. However, the previous research is inconclusive in terms of optimal feature representation and detection accuracy.

The primary objective of this thesis is to present state-of-the-art automated techniques for detecting and classifying malware using supervised learning algorithms. In particular, the focus is on two critical aspects of supervised learning-based malware detection: optimal feature representation and improved detection accuracy. Malware detection can be accomplished using two methods: static analysis, which extracts patterns without executing malware, and dynamic analysis, which captures behaviors through executing malware. This thesis focuses on static analysis instead of dynamic analysis because static analysis requires fewer computing resources. An additional benefit of static analysis is that present-day malware cannot evade it. To achieve the goals of this thesis, two new feature representations for static analysis are proposed. Furthermore, three customized ensembles are introduced to enhance malware detection accuracy, and their feasibility is experimentally demonstrated.  

The experiments incorporate customized malware data sets including Spyware, Adware, Scareware, and Android malware samples, and public malware data sets from Microsoft's having samples from nine distinct malware families. Artificially generated data sets are employed to mitigate class imbalance issues and represent inter-family and intra-family examples. Reverse engineering is performed to transform the data sets as feature data sets using both byte code and assembly language instructions. Further, existing and new feature representations along with various feature selection algorithms and feature fusion techniques are explored. To enhance detection accuracy, different decision theories from social choice theory, such as veto and consensus, are integrated into customized ensembles. The experimental results indicate that the proposed methods are capable of detecting known and zero-day malware. The proposed ensembles are also tested on the UCI public data sets, such as Forest CoverType, and the results demonstrate their effectiveness in classification. Further, these methods are designed to be portable and adaptable to different operating systems, and they can also be scaled for multi-class malware detection.