The cognitive radio network (CRN) concept has been proposed as a solution to the growing demand and underutilization of the radio spectrum. To improve the radio spectrum utilization, CRN technology allows the coexistence of licensed and unlicensed systems over the same spectrum. In an underlay spectrum sharing system, secondary users (SUs) transmit simultaneously with the primary users (PUs) in the same frequency band given that the interference caused by the SU to the PU remains below a tolerable interference limit. Besides the transmission power limitation, a secondary network is subject to distinct channel impairments such as fading and interference from the primary transmissions. Also, CRNs face new security threats and challenges due to their unique cognitive characteristics.This thesis analyzes the performance of underlay CRNs and underlay cognitive relay networks under spectrum sharing constraints and security constraints. Distinct SU transmit power policies are obtained considering various interference constraints such as PU outage constraint or PU peak interference power constraint.

The thesis is divided into an introduction and two research parts based on peer-reviewed publications. The introduction provides an overview of radio spectrum management, basic concepts of CRNs, and physical layer security. In the first research part, we study the performance of underlay CRNs with emphasis on a multiuser environment.In Part I-A, we consider a secondary network with delay-tolerant applications and analyze the ergodic capacity. Part I-B analyzes the secondary outage capacity which characterises the maximum data rate that can be achieved over a channel for a given outage probability. In Part I-C, we consider a secondary network with delay constrained applications, and derive expressions of the outage probability and delay-limited throughput. Part I-D presents a queueing model that provides an analytical tool to evaluate the secondary packet-level performance with multiple classes of traffic considering general interarrival and service time distributions. Analytical expressions of the SU average packet transmission time, waiting time in the queue, andtime spent in the system are provided.In the second research part, we analyze the physical layer security for underlay CRNs and underlay cognitive relay networks. Analytical expressions of the probability of non-zero secrecy capacity and secrecy outage probability are derived.Part II-A considers a single hop underlay CRN in the presence of multiple eavesdroppers (EAVs) and multiple SU-Rxs. In Part II-B, an underlay cognitive relay network in the presence of multiple secondary relays and multiple EAVs is studied.Numerical examples illustrate that it is possible to exploit the physical layer characteristics to achieve both security and quality of service in CRNs while satisfying spectrum sharing constraints.