Avllazagaj, ErinIn the pipeline of a cyber attack, the malicious actor will first gain a foothold in the target system through a malware. The malware detection is still a challenging problem, as the malware authors are constantly evolving their techniques to evade detection. Therefore, it is important for us to understand why that is the case and what can the defenders do to improve the detection of the malware. In this thesis, I explore the behavior of the malware in the real users’ machines and how it changes across different executions. I show that the malware exhibits more variability than benign samples and that certain actions are often more prone to variability than others. This is the first study that quantitatively analyzes the behavior of the malware in the wildI leverage an observation from the first project, where variability in the malware samples happens due to running privilege escalation exploits. The variability in behavior is due to the fact that the malware sometimes runs in non-privileged mode and tries to run an exploit to escalate its privileges. For these reasons, I propose a new methodology to systematically discover sensitive memory corruption targets that cause privilege escalation. At last, I explore the sensitive memory corruption targets in the Linux kernel. Specifically, I propose a methodology to systematically discover sensitive fields in the Linux kernel that, when corrupted, lead the system into an exploitable state. This system, called SCAVY, is based on a novel definition of the exploitable state that allows the attacker to read and write into files and memory locations that they would normally. SCAVY explores the exploitable states based on the threat model of a local unprivileged attacker with the ability to issue system calls and with the capability to read/write into a limited location in the kernel memory. The framework revealed that there are 17 sensitive fields across 12 Linux kernel C structs that, when overwritten with the correct value, lead the system into an exploitable state. In this definition, unlike prior work, I consider the system to be in an exploitable state when the weird machine allows the attacker to read and/or write into files and memory locations that they would normally not be able to. This state can be used to write into sensitive files such as //etc//passwd where the exploit author can create a new root account on the vulnerable host and log in as that. Additionally, if the attacker can read unreadable files such as //etc//shadow they can leak passwords of root accounts, de-hash them and log in as the root account. I utilize these targets to develop 6 exploits for 5 CVE vulnerabilities. I also demonstrated the severity of these fields and the applicability of the exploitable state by exploiting CVE-2022-27666. I overwrote the f mapping pointer in struct file and caused a write into //etc//passwd. Unlike the original exploit, ours didn’t need to break KASLR, modify global variables or require support of FUSE-fs from the vulnerable host. This makes our methodology more extensible and more stable, since the exploit requires fewer corruption in the kernel memory and it doesn’t rely on the need to have the addresses of the kernel’s symbols for calculating the KASLR offset. Additionally, our exploit doesn’t modify global variables, which makes it more stable and less likely to crash the kernel, during its runtime. Our findings show that new memory corruption targets can change the security implications of vulnerabilities, urging researchers to proactively discover memory corruption targets.enDissertationComputer engineeringInformation sciencedetectionexploitkernelmalware