Cognitive radio technology is a new revolutionary communication paradigm which allows flexible access to spectrum resources and leads to efficient spectrum sharing. Recent studies have shown that cognitive radio is a promising approach to improve efficiency of spectrum utilization, because wireless users are capable of accessing the spectrum in an intelligent and adaptive manner. The theory of cognitive radio is however still immature to fully understand its broader impacts on the design of future wireless networks. This dissertation contributes to the advancement of cognitive radio technology by analyzing wireless users' interaction in a network and developing game-theoretic frameworks to suppress selfish and malicious behaviors, with the goal to improve system performance by stimulating selfish users and enhance network security against malicious users.

We first develop a cheat-proof repeated spectrum sharing game, which provides the incentive for selfish users to cooperate with each other and reveal their private information truthfully. We propose specific cooperation rules based on the maximum total throughput and proportional fairness criteria, and investigate the impact of spectrum sensing duration on system performance.

We also consider the situation where a group of selfish users collude for higher payoffs. We propose a novel multi-winner spectrum auction framework which did not exist in auction literature, and develop collusion-resistant auction mechanisms to suppress collusive behavior. In addition, we apply the semi-definite programming relaxation to significantly reduce the complexity of algorithms.

When malicious users are taken into consideration, we apply game-theoretic tools to suppress potential malicious behavior in cognitive radio networks. Specifically, we model the anti-jamming defense as a zero-sum game, and derive the optimal strategy for secondary users to execute in face of jamming threats. Moreover, we propose learning schemes for secondary users to gain knowledge of adversaries.

Finally, we consider security countermeasures against eavesdroppers, and propose a cooperative paradigm that primary users improve secrecy with the help of trustworthy secondary users. We derive the achievable pair of primary users' secrecy rate and secondary users' transmission rate under various circumstances, and model the interaction between primary users and secondary users as a Stackelberg game.