To achieve better connectivity in future communication networks, the deployment of different types of access points (APs) is underway. APs are expected to be equipped with selforganizing capabilities to reduce costs. Moreover, due to the spectrum crunch, frequency reuse among the deployed APs is inevitable, exacerbating the problem of inter-cell interference (ICI). Therefore, ICI mitigation in self-organizing networks (SONs) is commonly identified as a key radio resource management mechanism to enhance performance. To this end, this paper proposes a novel solution for the uncoordinated channel and power allocation problems. Based on the multi-armed bandits (MAB) framework, the proposed technique does not require any communication between the APs. The case of varying channel rewards across APs is considered. In contrast to previous work on channel allocation using the MAB framework, APs are permitted to choose multiple channels for transmission. Moreover, nonorthogonal multiple access is used, allowing multiple APs to access each channel simultaneously. This results in an MAB model with varying channel rewards, multiple plays and nonzero reward on collision. The proposed algorithm has an expected regret in the order of O(log 2 T), with extensive numerical results revealing it significantly outperforms a well-known baseline algorithm in terms of energy efficiency.