MSc thesis supervision: A Systematic Study of Real-World Exploit Complexity in Android Security Bulletins

Abstract

Despite its widespread use, prior research has shown that CVSS vulnerability severity evaluations can suffer from inconsistencies between its human evaluators and can fail to reflect contextual factors present in real-world systems. However, there has been limited research on how these inconsistencies manifest specifically in Android environments, especially in physical devices, instead mainly utilizing theoretical models or emulated environments. This thesis investigates that gap by experimentally testing on physical Android devices, and examining how well CVSS aligns with real-world, practical exploit complexity for vulnerabilities disclosed in the Android Security Bulletin.

A custom tool was made where all CVEs (vulnerabilities) from the Android Security Bulletin were collected, and a targeted selection was made using a custom prioritization system designed to identify likely exploitable CVEs compatible with practical reproduction. Selected CVEs were then reproduced on real devices to evaluate how CVSS reflects exploitability in practical environments in terms of difficulty, behavior, and official scoring.

The results revealed that a high percentage of the practically tested CVEs showed a discrepancy in the official scoring in real-world environments across multiple metrics, both in exploitability and impact as defined by CVSS. It also showed that discrepancies may stem from contextual factors like the granularity of privilege, the permission architecture of Android, and ambiguous metric interpretation. Where vulnerabilities saw justification in an adjusted score, most saw a decrease in severity. The trends showed an imbalance in the types of vulnerability and highlighted varying exploitability between them. In addition, other variables such as the neglect of auxiliary bug paths in CVSS scoring, inconsistent behavior between Android versions and devices, and a potential case of residual attack surface were discovered.

These findings highlight the limitation of CVSS in capturing real-world risk in Android environments and the need for practical empirical testing. Without it, severity scores risk being misleading or lose precision, which can lead to worse patch prioritization and vulnerability management.

My role as supervisor

I was the student's university supervisor at IDA, with Ulf Kargén as examiner. The thesis was carried out in cooperation with Sectra Communications AB. I reviewed and steered the scientific writing up to the final thesis.