In recent years, the innovation of multiple-input multiple-output (MIMO) has completely changed the face of wireless communication system. Especially, the concept of MIMO is evolved to fulfill the demand of multimedia services. However, challenges have risen to the system provider for maintaining the quality of service (QoS) and reliability of MIMO system. Additionally, very recently, cooperative communication has been proposed in order to face and deal with the shortcomings of MIMO system. In cooperative communication, the addition of relay between source and destination enhances the QoS and reliability of wireless system. The main purpose of this paper is to derive the closed form analytical expressions of symbol error rate (SER) of M-ary phase shift keying (MPSK) for MIMO decode-and forward (DF) relay system. Besides, SER derivation is based on moment generating function (MGF) of the overall system. In our case, we only deal with a downlink MIMO DF relay system. Here, we consider an $n_t$-antenna at the source $S$, an $n_r$-antenna at the relay $R$ and an $n_d$-antenna at the destination $D$, where the propagated signal from the source passes through keyhole and correlated channel. Further, the same orthogonal space time block codes (OSTBC) is applied to source-destination, source-relay and relay-destination link to exploit the maximal diversity gain. Furthermore, we analyze the effects of MPSK modulation on MIMO DF system under different values of correlation coefficient by combining the different antenna pairs at different nodes. Beside this, we compare keyhole and no keyhole MIMO relay systems with fix value of correlation coefficient and antenna pairs. With the use of cooperative scheme, effects of keyhole on MIMO DF architecture can be minimized to certain level. Eventually, our simulation analysis excellently matches with analytical results, which is sufficient to validate our work.