Non-intrusive hardware-enhanced anomaly detection systems for embedded devices

The integrity of billions of embedded devices is threatened by adversarial attacks, environmental factors, architectural flaws and programming bugs. Conventional defences are largely class-specific: signature checking for adversarial attacks; error-checking memory for environmental faults; OS-level patches for architectural flaws; and static analysis for software defects. In contrast, anomaly detection provides threat-agnostic coverage, recognising deviations from a known safe behaviour or behavioural specification, independent of the root cause. This thesis explores hardware-enhanced anomaly detection in embedded devices, aiming to overcome challenges arising from the behavioural complexity of programs, the large amount of trace produced by processors and the non-linearity of trace data, facilitating non-intrusiveness (to avoid harmful delays), responsiveness (for timely detection), determinism (for consistency) and flexibility (for specialised use-cases), a set of traits identified in this work as desirable in a monitoring system but often missing in existing solutions. This work investigates non-security-related CPU registers for security purposes, measuring the suitability of hardware performance counters for anomaly detection and general-purpose registers for program execution partitioning. The first technical chapter evaluates machine learning algorithms (one-class SVM, local outlier factor, isolation forest, n-grams) on low-level CPU trace data, revealing the need for a trace reduction method, facilitating consistent data collection. Subsequent chapters introduce a HW/SW co-design (establishing the trace reduction requirement for non-intrusive monitoring) and a Fine-grained CPU-state-based Trace Qualification (FCSTQ) method, which partitions execution depending on selected bits of values constituting the current CPU-state, such as program counter, instruction and general purpose registers. The first study compares FCSTQ against periodic and every n-instruction sampling, showing it collects 8.3 and 9.6 times more consistent data, respectively. The second study evaluates automatic detection of regular, infrequent CPU-state events and contrasts it with control-flow-graph-based partitioning, which tends to yield irregular triggers. FCSTQ achieved an 8.6 times reduction in trace volume and an 11.2 times smaller worst-case observational gap. Both studies used the EEMBC Automotive 1.1 benchmark, chosen for its prior adoption in related work and modular composition. The final technical chapter amends the FCSTQ with programmable actions and dedicated memory, forming the Anomaly-detection-oriented Micro-scale Processing (AMP) system, allowing FCSTQ conditions to perform arithmetic operations and depend on their outcomes. This novel mechanism is capable of verifying adherence to a behavioural specification in hardware through independent background computation, as well as leveraging the trace reduction to enable non-intrusive anomaly detection in software, bridging a gap between performant but rigid hardware-based monitoring methods and relatively slow but flexible software-based monitoring methods.