Q-Learning based Dynamic Electromagnetic Spectrum Management: Design, Modeling, and Performance Evaluation

Abstract

With the rapid development of wireless communication technologies and wide application of clustered devices (such as UAV swarms), the demand for electromagnetic spectrum resources has surged, while traditional static spectrum management struggles to adapt to dynamic complex electromagnetic environments, leading to low spectrum utilization, frequent inter-device interference, and difficulty meeting real-time communication needs. To address these issues, this study proposes a dynamic electromagnetic spectrum management method based on improved Q-Learning. A Markov Decision Process (MDP) model is constructed, with the state space defined by real-time spectrum channel occupancy, user equipment SINR, and device spatial coordinates; the action space includes channel selection and transmit power adjustment; the reward function balances spectrum utilization, interference reduction, and latency minimization. The Q-Learning algorithm is enhanced with dynamic learning rate adjustment and priority experience replay to optimize key parameters (channel switching latency, SINR threshold, transmit power stability). Comparative experiments with traditional static spectrum allocation show the proposed method improves spectrum utilization by 32.5%, reduces interference rate by 41.2%, and shortens switching latency by 28.8% on average. Additionally, integrating Markov decision logic into OFDM’s subcarrier allocation and power control improves its adaptability to dynamic spectra, enhancing spectral efficiency by 15.7% compared to traditional OFDM.