Development of novel reinforcement learning-based optimizer to impede tumor growth via radiochemotherapy

Current cancer treatment strategies prioritize algorithmic robustness and efficiency but frequently neglect critical aspects of patient safety and comfort. These approaches typically rely on chemotherapy-based mathematical models optimized solely for short-term tumor reduction, disregarding the broader impact on patient health. This study introduces a patient-centered approach to optimize cancer treatment, balancing treatment efficacy and toxicity. The proposed research method incorporates both radiation therapy and chemotherapy simultaneously in the form of ordinary differential equations (ODE)-based mathematical dynamics. These updated dynamics are then utilized to propose a novel control mechanism that integrates nonlinear sliding mode control (SMC) with reinforcement learning-based proximal policy optimization (PPO) algorithm. Conventional sliding mode control (SMC) algorithm is first modified by replacing its signum function-based switching control with a sigmoid function to address issues like chattering and transients in treatment control. This smooth SMC is then incorporated within the framework of PPO to dynamically adjust treatment schedules, reduce drug and radiation dosages, smooth administration of treatment dosages, and enhance patient health indicators. Results showed that the proposed hybrid PPO method effectively lowered chemotherapy and radiotherapy dosages while maintaining tumor suppression, minimizing treatment toxicity, and improving immune cell recovery. In quantitative comparisons, the proposed PPO algorithm reduced baseline dosages by up to 76.8% for chemotherapy and 66% for radiotherapy and achieved tumor suppression 5.67% faster than conventional multi-input optimization methods. It also lowered cumulative treatment intensity by over 92%, demonstrating a substantial enhancement in patient safety. The methodological originality of this study lies in integrating nonlinear smooth SMC with reinforcement learning-based PPO within a patient-centered ODE modeling framework that jointly represents radiotherapy, chemotherapy, tumor dynamics, immune cell dynamics, healthy cell preservation, and health indicator state. The proposed framework provides a relevant, toxicity-aware computational approach for radio-chemotherapy dosage optimization, demonstrating lower simulated treatment intensity while maintaining tumor suppression under stated model assumptions. As a pre-clinical computational proof of concept, this work establishes a robust and interpretable basis for future treatment-planning studies, subject to retrospective clinical validation, patient-specific parameterization, and prospective safety evaluation.