5G and beyond communication systems are envisaged to fulfill three key promises that enable novel use cases and applications such as telemedicine, augmented reality/virtual reality (AR/VR), smart manufacturing, autonomous vehicles (AVs), etc. These three key promises are i) Enhanced mobile broadband (eMBB), ii) Ultra-reliable low latency Communications (URLLC), and iii) Massive machine-type communications (mMTC). In other words, 5G is required to achieve key performance indicators (KPIs) in terms of low latency, massive device connectivity, consistent quality of service (QoS), and high security. For instance, user bit-rates up to 10 Gbps and round-trip times (RTTs) as small as 1–10 ms are demanded in specific application scenarios in 5G. Toward achieving the 5G key promises, it is essential to utilize the capacity of all sorts of communications networks (terrestrial, space, aerial) and supporting technologies (SDN, NFV, etc.) simultaneously, leading to the so-called hybrid communication networks as opposed to the traditional stand-alone ones. This signifies the importance of a seamless integration and configuration policy tailored to specific use cases and QoS requirements of 5G and beyond services and will spawn several challenging design and optimization problems from the control and management to the physical layer of next-generation systems. In this thesis, we will address such critical problems in the course of 9 chapters.

In the second chapter, we study the benefits of incorporating trust into decision-making for resource provisioning in next-generation communications networks. In this regard, we study the trust-aware service chain embedding problem for enhancing the reliability of virtual network function (VNF) placement on the trusted infrastructure. The problem of placing the VNFs onthe NFV infrastructure (NFVI) and establishing the routing paths between them, according to the service chain template, is termed SFC embedding. The objectives and constraints for the optimization problem formulation of SFC embedding may vary depending on the corresponding network service. We introduce the notion of trustworthiness as a measure of security in SFC embedding and thus network service deployment. We formulate the resulting trust-aware SFC embedding problem as a Mixed Integer Linear Program (MILP). We relax the integer constraints to reduce the time complexity of the MILP formulation and obtain a Linear Program (LP). We investigate the trade-offs among the two formulations, seeking to strike a balance between results accuracy and time complexity.

The space-air-ground integrated network (SAGIN) offers potential benefits that are not possible otherwise, including global coverage, low latency, and high reliability. On the other hand, the heterogeneity of the integrated network with non-unified interfaces, and the diversity of 5G use cases with large-scale applications highlight the need for a unified management structure and a dynamic resource allocation policy that are both scalable and flexible enough to handle the increasing complexity. In the third chapter, on one hand, we optimize the integration of the hybrid network by deployment of satellite gateways on the ground segment of the network to ensure proper connection between the layers with minimum latency, and on the other hand, we aim at providing a seamless management and control scheme for the hybrid network utilizing the capacities of the supportive technologies, software-defined networking (SDN) and network function virtualization (NFV); In particular, we study the problem of SDN controller placement with the goal maximizing the reliability of the hybrid network.

In the fourth chapter, we propose trust as a metric to measure the trustworthiness of the FL agents and thereby enhance the security of the FL training. We first elaborate on trust as a security metric by presenting a mathematical framework for trust computation and aggregation within a multi-agent system. We then discuss how this framework can be incorporated within an FL setup introducing the trusted FL algorithm for both centralized and decentralized FL. Next, we propose a framework for decentralized FL in UAV-enabled networks which involves the placement of the UAVs while ensuring the connectivity of the network of deployed UAVs.

We dedicate the remaining chapters to studying the novel design problems and the key technologies for the physical layer of next-generation wireless systems with an emphasis on millimeter-wave communications, massive MIMO, and hybrid beamforming. We introduce a novel antenna configuration called twin-ULA (TULA) and its composite configurations to generate sharp beams with maximal and uniform gain. We introduce a novel beam alignment technique to maximize the utility of transmission in the presence of multipath, efficiently utilize reconfigurable intelligent surfaces (RIS) to enhance mmWave coverage in urban environments, and synchronize and calibrate in distributed massive MIMO networks for 6G systems, where the synchronization involves the carrier frequency offset estimation and compensation, and the calibration involves mitigating reciprocity mismatches in digital and analog RF chains of the access points (APs) implementing hybrid beamforming, enabling efficient downlink channel estimation.