Enhancement of Large Language Models Driving Knowledge for Practical Autonomous Driving Decision Making

Theoretical driving knowledge and hazard perception capability are essential for large language models (LLMs) to be qualified for driving or assisting autonomous vehicles. Our prior study found that the mainstream LLMs with relatively small size suitable for onboard or edge deployment failed driving theory tests. That raised several important research questions on application of LLMs for safety critical autonomous driving tasks: how to enhance the theoretical driving knowledge of LLMs and how to assess its impact on driving decision-making and safety in realistic scenarios. In this paper we first propose a novel question guided multimodal graph based retrieval-augmented generation (QGM-GRAG) approach to address the first question. Structured textual rules and visual traffic sign semantics are integrated to create a unified graph-based representation. A joint representation that fuses paragraph level embeddings with question aware signals is generated by LLMs for accurate and interpretable retrieval. Then we propose a general scenario based evaluation framework to address the second question, by investigating the connection between the driving theory knowledge and practical driving decision-making performance. In the framework theory-test questions are enhanced and converted to realistic driving scenarios. Prompts are designed for LLMs to generate driving decisions. The decisions are evaluated automatically by a GPT-4o based agent across multiple dimensions (such as situation awareness and driving safety). Experiments demonstrate that the proposed RAG approach can achieve more than 10% improvement on driving theory tests. In addition, results from the evaluation framework show that the driving knowledge has significant impact on driving decisions and the QGM-GRAG approach can help improve driving performance by more than 13%. Furthermore, experiments on the real-world LingoQA dataset show an average improvement of 44.87%, confirming the practical applicability of our pipeline. These results validate the effectiveness and robustness of the proposed RAG approach and evaluation framework.