We examine the problem of designing an auction-based market mechanism for dynamic spectrum sharing when there are multiple sellers and multiple buyers. We assume that the sellers are selfish players and focus on an optimal auction mechanism that maximizes the expected payoff or profit of the seller.

First, we study the interaction among homogeneous buyers of the spectrum as a noncooperative game and show the existence of a symmetric mixed strategy Nash equilibrium (SMSNE). We investigate the uniqueness of the SMSNE in some special cases and discuss the convergence to the unique SMSNE.

Second, we prove that there exists an incentive for risk neutral sellers of the spectrum to cooperate in order to maximize their expected profits at the SMSNEs of buyers' noncooperative game. This is done by modeling the interaction among the sellers as a cooperative game and demonstrating that the core of the cooperative game is nonempty. We show that there exists a way for the sellers to share the profits in a such manner that no subset of sellers will have an incentive to deviate or power to increase their expected profits by deviating. We also introduce the algorithms for achieving any profit sharing in the core.

Finally, we introduce an optimal auction mechanism in which the spectrum bands in multiple regions are sold simultaneously and the buyers are simple-minded in the sense that each buyer wants to buy the same number of frequency bands only the regions where they operate.