Deep Reinforcement Learning for Wireless Communications and Networking - Theory, Applications and Implementation

Deep Reinforcement Learning for Wireless Communications and Networking
 
Comprehensive guide to Deep Reinforcement Learning (DRL) as applied to wireless communication systems
 
Deep Reinforcement Learning for Wireless Communications and Networking presents an overview of the development of DRL while providing fundamental knowledge about theories, formulation, design, learning models, algorithms and implementation of DRL together with a particular case study to practice. The book also covers diverse applications of DRL to address various problems in wireless networks, such as caching, offloading, resource sharing, and security. The authors discuss open issues by introducing some advanced DRL approaches to address emerging issues in wireless communications and networking.
 
Covering new advanced models of DRL, e.g., deep dueling architecture and generative adversarial networks, as well as emerging problems considered in wireless networks, e.g., ambient backscatter communication, intelligent reflecting surfaces and edge intelligence, this is the first comprehensive book studying applications of DRL for wireless networks that presents the state-of-the-art research in architecture, protocol, and application design.
 
Deep Reinforcement Learning for Wireless Communications and Networking covers specific topics such as:
* Deep reinforcement learning models, covering deep learning, deep reinforcement learning, and models of deep reinforcement learning
* Physical layer applications covering signal detection, decoding, and beamforming, power and rate control, and physical-layer security
* Medium access control (MAC) layer applications, covering resource allocation, channel access, and user/cell association
* Network layer applications, covering traffic routing, network classification, and network slicing
 
With comprehensive coverage of an exciting and noteworthy new technology, Deep Reinforcement Learning for Wireless Communications and Networking is an essential learning resource for researchers and communications engineers, along with developers and entrepreneurs in autonomous systems, who wish to harness this technology in practical applications.

List of contents

Notes on Contributors xiii
 
Foreword xiv
 
Preface xv
 
Acknowledgments xviii
 
Acronyms xix
 
Introduction xxii
 
Part I Fundamentals of Deep Reinforcement Learning 1
 
1 Deep Reinforcement Learning and Its Applications 3
 
1.1 Wireless Networks and Emerging Challenges 3
 
1.2 Machine Learning Techniques and Development of DRL 4
 
1.2.1 Machine Learning 4
 
1.2.2 Artificial Neural Network 7
 
1.2.3 Convolutional Neural Network 8
 
1.2.4 Recurrent Neural Network 9
 
1.2.5 Development of Deep Reinforcement Learning 10
 
1.3 Potentials and Applications of DRL 11
 
1.3.1 Benefits of DRL in Human Lives 11
 
1.3.2 Features and Advantages of DRL Techniques 12
 
1.3.3 Academic Research Activities 12
 
1.3.4 Applications of DRL Techniques 13
 
1.3.5 Applications of DRL Techniques in Wireless Networks 15
 
1.4 Structure of this Book and Target Readership 16
 
1.4.1 Motivations and Structure of this Book 16
 
1.4.2 Target Readership 19
 
1.5 Chapter Summary 20
 
References 21
 
2 Markov Decision Process and Reinforcement Learning 25
 
2.1 Markov Decision Process 25
 
2.2 Partially Observable Markov Decision Process 26
 
2.3 Policy and Value Functions 29
 
2.4 Bellman Equations 30
 
2.5 Solutions of MDP Problems 31
 
2.5.1 Dynamic Programming 31
 
2.5.1.1 Policy Evaluation 31
 
2.5.1.2 Policy Improvement 31
 
2.5.1.3 Policy Iteration 31
 
2.5.2 Monte Carlo Sampling 32
 
2.6 Reinforcement Learning 33
 
2.7 Chapter Summary 35
 
References 35
 
3 Deep Reinforcement Learning Models and Techniques 37
 
3.1 Value-Based DRL Methods 37
 
3.1.1 Deep Q-Network 38
 
3.1.2 Double DQN 41
 
3.1.3 Prioritized Experience Replay 42
 
3.1.4 Dueling Network 44
 
3.2 Policy-Gradient Methods 45
 
3.2.1 REINFORCE Algorithm 46
 
3.2.1.1 Policy Gradient Estimation 46
 
3.2.1.2 Reducing the Variance 48
 
3.2.1.3 Policy Gradient Theorem 50
 
3.2.2 Actor-Critic Methods 51
 
3.2.3 Advantage of Actor-Critic Methods 52
 
3.2.3.1 Advantage of Actor-Critic (A2C) 53
 
3.2.3.2 Asynchronous Advantage Actor-Critic (A3C) 55
 
3.2.3.3 Generalized Advantage Estimate (GAE) 57
 
3.3 Deterministic Policy Gradient (DPG) 59
 
3.3.1 Deterministic Policy Gradient Theorem 59
 
3.3.2 Deep Deterministic Policy Gradient (DDPG) 61
 
3.3.3 Distributed Distributional DDPG (D4PG) 63
 
3.4 Natural Gradients 63
 
3.4.1 Principle of Natural Gradients 64
 
3.4.2 Trust Region Policy Optimization (TRPO) 67
 
3.4.2.1 Trust Region 69
 
3.4.2.2 Sample-Based Formulation 70
 
3.4.2.3 Practical Implementation 70
 
3.4.3 Proximal Policy Optimization (PPO) 72
 
3.5 Model-Based RL 74
 
3.5.1 Vanilla Model-Based RL 75
 
3.5.2 Robust Model-Based RL: Model-Ensemble TRPO (ME-TRPO) 76
 
3.5.3 Adaptive Model-Based RL: Model-Based Meta-Policy Optimization (mb-mpo) 77
 
3.6 Chapter Summary 78
 
References 79
 
4 A Case Study and Detailed Implementation 83
 
4.1 System Model and Problem Formulation 83
 
4.1.1 System Model and Assumptions 84
 
4.1.1.1 Jamming Model 84
 
4.1.1.2 System Operation 85
 
4.1.2 Problem Formulation 86
 
4.1.2.1 State Space 86
 
4.1.2.2 Action Space 87
 
4.1.2.3 Immediate Reward 88
 
4.1.2.4 Optimization

About the author

Dinh Thai Hoang, Ph.D., is a faculty member at the University of Technology Sydney, Australia. He is also an Associate Editor of IEEE Communications Surveys & Tutorials and an Editor of IEEE Transactions on Wireless Communications, IEEE Transactions on Cognitive Communications and Networking, and IEEE Transactions on Vehicular Technology. Nguyen Van Huynh, Ph.D., obtained his Ph.D. from the University of Technology Sydney in 2022. He is currently a Research Associate in the Department of Electrical and Electronic Engineering, Imperial College London, UK. Diep N. Nguyen, Ph.D., is Director of Agile Communications and Computing Group and a member of the Faculty of Engineering and Information Technology at the University of Technology Sydney, Australia. Ekram Hossain, Ph.D., is a Professor in the Department of Electrical and Computer Engineering at the University of Manitoba, Canada, and a Fellow of the IEEE. He co-authored the Wiley title Radio Resource Management in Multi-Tier Cellular Wireless Networks (2013). Dusit Niyato, Ph.D., is a Professor in the School of Computer Science and Engineering at Nanyang Technological University, Singapore. He co-authored the Wiley title Radio Resource Management in Multi-Tier Cellular Wireless Networks (2013).

Product details

• Authors: Dinh Thai Hoang, Dinh Thai (University of Technology Sydney Hoang, Ekram Hossain, Hossain Ekram, Nguyen Van Huynh, Diep N Nguyen, Diep N. Nguyen, Dusit Niyato, Niyato Dusit
• Publisher: Wiley, John and Sons Ltd
• Languages: English
• Product format: Hardback
• Released: 25.07.2023
• EAN: 9781119873679
• ISBN: 978-1-119-87367-9
• No. of pages: 288
• Subjects: Natural sciences, medicine, IT, technology > Technology > Electronics, electrical engineering, communications engineering; Informatik, Künstliche Intelligenz, Artificial Intelligence, computer science, drahtlose kommunikation, Mobile & Wireless Communications, Electrical & Electronics Engineering, Elektrotechnik u. Elektronik, Networking / Security, Netzwerke / Sicherheit