Cellular networks are facing serious economic and ecological challenges due to the exponential increase in mobile traffic. As a promising direction, mobile operators are equipping base stations with renewable energy and battery systems along with energy efficiency techniques. In this paper, we study cellular networks equipped with batteries and powered by renewable energy sources and the Smart Grid. We exploit reinforcement learning to minimize the grid energy cost and maximize the users’ satisfaction considering variable price of grid energy, traffic variation and renewable energy generation. In contrast to existing studies, we take into consideration both heterogeneity of users and degradation of battery. We propose a Q-learning algorithm that decides the best number of active radio resources considering two cases: with and without battery constraints. Simulation results highlight the importance of imposing constraints on the battery operation. When the battery size is large enough, the battery life is extended with negligible degradation in system performance. In addition, while imposing constraints on the battery may lead to performance degradation on the short term, this is compensated on the long term as shown by simulating the system over one year.