Leveraging malicious behavior traces from volatile memory using machine learning methods for trusted unknown malware detection in Linux cloud environments

Most organizations today use cloud-computing environments and virtualization technology. Linux-based clouds are the most popular cloud environments among organizations, and thus have become the target of cyber-attacks launched by sophisticated malware. Existing malware detection solutions for Linux-based VMs are installed and operated on the VM itself and are considered untrusted since malware can detect, interfere with, and even evade them. Thus, Linux cloud-based environments remain exposed to various malware-based attacks. This paper presents the first trusted framework for detecting unknown malware in Linux VM cloud-environments. Our framework acquires volatile memory dumps from the inspected VM by querying the hypervisor in a trusted manner and overcoming malware's ability to detect the security mechanism and evade detection. Then, using machine-learning algorithms we leverage informative traces (our 171 proposed features) from different parts of the VM's volatile memory. The framework was evaluated in seven rigorous experiments, on a total of 21,800 volatile memory dumps taken from two widely used virtual servers (10,900 from each server) during the execution of a diverse yet representative collection of benign and malicious Linux applications. Notably, the results show that our proposed framework can accurately (with high TPRs and low FPRs): (a) detect unknown malware (b) detect new unknown malware from unseen malware categories, which is a critical ability for coping with new malware trends and phenomena; (c) categorize an unknown malware by its attack category; (d) detect unknown malware on an unknown virtual-server; and lastly (e) detect fileless malware, a critical capability demonstrating the ability to detect substantially different attack modus operandi.