Global Path Planning for Unmanned Surface Vehicles based on Adaptive Parameter Q-Learning Algorithm

Abstract

Global path planning is one of the core issues in the field of Unmanned Surface Vehicle (USV) navigation. As a classic method in reinforcement learning, the Q-Learning algorithm is widely applied to USV global path planning. However, the traditional Q-Learning algorithm typically employs a fixed learning rate, greedy rate, and discount factor, which leads to issues such as redundant exploration in the late training stage and slow convergence speed. To address these limitations, this paper proposes an adaptive parameter Q-Learning path planning algorithm. The proposed algorithm dynamically adjusts the greedy rate to balance exploration and exploitation, adaptively optimizes the learning rate based on the temporal difference (TD) error, and dynamically adjusts the discount factor by combining the distance between the current state and the destination. These improvements enhance both the convergence efficiency and path planning performance of the algorithm. Comparative experiments between the improved algorithm and the traditional Q-Learning algorithm were conducted in a simulated water environment. Experimental results demonstrate that the improved algorithm increases the training convergence speed by over 30% and shortens the optimal path length by 11%, indicating a significant improvement in algorithm performance. This study provides a more efficient algorithmic scheme for USV global path planning and holds great significance for enhancing the real-time performance, reliability, and environmental adaptability of USV navigation.