In this paper, we propose a sleep strategy for energy-efficient 5G Base Stations (BSs) with multiple Sleep Mode (SM) levels to bring down energy consumption. Such management of energy savings is coupled with managing the Quality of Service (QoS) resulting from waking up sleeping BSs. As a result, a tradeoff exists between energy savings and delay. Unlike prior work that studies this problem for binary state BS (ON and OFF), this work focuses on multi-level SM environment, where the BS can switch to several SM levels. We propose a Q-Learning algorithm that controls the state of the BS depending on the geographical location and moving velocity of neighboring users in order to learn the best policy that maximizes the tradeoff between energy savings and delay. We evaluate the performance of our proposed algorithm with an online suboptimal algorithm that we introduce as well. Results show that the Q-Learning algorithm performs better with energy savings up to 92% as well as better delay performance than the heuristic scheme.