Book Description

A new era of wireless networks is evolving, thanks to the significant advances in communications and networking technologies. In parallel, wireless services are witnessing a tremendous change due to increasingly heterogeneous and stringent demands, whose quality of service requirements are expanding in several dimensions, putting pressure on mobile networks. Examples of those services are augmented and virtual reality, as well as self-driving cars. Furthermore, many physical systems are witnessing a dramatic shift into autonomy by enabling the devices of those systems to communicate and transfer control and data information among themselves. Examples of those systems are microgrids, vehicles, etc. As such, the mobile network indeed requires a revolutionary shift in the way radio resources are assigned to those services, i.e., RRM. In RRM, radio resources such as spectrum and power are assigned to users of the network according to various metrics such as throughput, latency, and reliability. Several methods have been adopted for RRM such as optimization-based methods, heuristics and so on. However, these methods are facing several challenges such as complexity, scalability, optimality, ability to learn dynamic environments. In particular, a common problem in conventional RRM methods is the failure to adapt to the changing situations. For example, optimization-based methods perform well under static network conditions, where an optimal solution is obtained for a snapshot of the network. This leads to higher complexity as the network is required to solve the optimization at every time slot. Machine learning constitutes a promising tool for RRM with the aim to address the conflicting objectives, i.e., KPIs, complexity, scalability, etc. In this thesis, we study the use of reinforcement learning and its derivatives for improving network KPIs. We highlight the advantages of each reinforcement learning method under the studied network scenarios. In addition, we highlight the gains and trade-offs among the proposed learning techniques as well as the baseline methods that rely on either optimization or heuristics. Finally, we present the challenges facing the application of reinforcement learning to wireless networks and propose some future directions and open problems toward an autonomous wireless network. The contributions of this thesis can be summarized as follows. First, reinforcement learning methods, and in particular model-free Q-learning, experience large convergence time due to the large state-action space. As such, deep reinforcement learning was employed to improve generalization and speed up the convergence. Second, the design of the state and reward functions impact the performance of the wireless network. Despite the simplicity of this observation, it turns out to be a key one for designing autonomous wireless systems. In particular, in order to facilitate autonomy, agents need to have the ability to learn/adjust their goals. In this thesis, we propose transfer in reinforcement learning to address this point, where knowledge is transferred between expert and learner agents with simple and complex tasks, respectively. As such, the learner agent aims to learn a more complex task using the knowledge transferred from an expert performing a simpler (partial) task.