In the last few decades, the innovation of Wireless communication has been developed very fast in order to improve the performance of communication systems. Especially,a concept of Multiple Input and Multiple Output (MIMO) is purposed to fulfill a high data rate service such as high quality video conference. However, a limitation in size and power of mobile device in the latest version of cellular system i.e. third Generation (3G) and fourth Generation (4G) causes difficulty to implement MIMO on mobile unit. Hence, Cooperative communication has been created to operate as a virtual MIMO in modern Wireless communication. In the main purpose of this paper, the performance analysis of Hybrid Decode-Amplify and Forward (HDAF) Cooperative communication using Orthogonal Space-Time Block Code (OSTBC) is derived in term of Symbol Error Rate (SER) against Signal-to-Noise Ratio (SNR) when the system applies different type of modulation techniques. Additionally, the error performance is derived base on Moment Generating Function (MGF) of the Rayleigh fading channel. In our thesis, we only cope with the down link of the system. Assuming that multiple antennas can be equipped into a transmitter which operates as base station in cellular system, mobile unit can be equipped with only single antenna due to the size limitation which operates as relays and destination. Moreover, the receiver uses Maximum Ratio Combining (MRC) to receive the transmitted signal. In the first step, we derive the performance of single relay system in order to show a better performance when increase the number of antennas at the transmission. In the second case, the performance of multiple relays is derived in order to express the behavior of the system when the number of relay is increased. Finally, the saturated number of relay can be investigated by the concept of relaying gain.

In this thesis, the benefit of HDAF cooperative communication using multiple antennas at transmitter with single antenna at the relay and receiver are investigated. The evaluated equations of the model are implemented in MATLAB in order to observe its system performance. The conclusion from the numerical analysis shows that the multiple antennas at the transmitter with only one antenna at the relay and the destination improve the performance of the system when compared to convention hybrid protocol.Furthermore, the multiple relay system is study by extending the knowledge from the single relay system. The result shows that the multiple relays improve the performance of HDAF significantly if a number of relays are increased. However, the relaying gain shown that relaying gain is limited by some number of the relay. This means that even though a large number of relays are added into the system, they will not increase the performance of the HDAF in the sense that the SEP of HDAF compared to DF and AF. The requirednumber or optimal number of the relays in HDAF system is the smallest number which makes relaying gain still approaches its maximum values