With the advent of new multimedia applications the demand for in time delivery of data is increased as compared to the reliability. Usually the Transport Layer Protocols, User Datagram Protocol (UDP) and Transmission Control Protocol (TCP) are used to transfer the data over the IP based network like Internet. TCP provides a reliable mechanism to transfer the data but its reliable mechanism results in increase in delay. UDP lacks in providing any acknowledgment mechanism and it does not provide any congestion control mechanism also. However the unreliable behavior of UDP results in less delay in data transfer. Now a days one of the important issues is the Quality of Service (QoS) assurance as the behavior of transport layer protocols can affect the QoS. So in order to avoid these issues some new transport layer protocols have been developed by Internet Engineering Task Force (IETF). Two important transport layer protocols, Datagram Congestion Control Protocol (DCCP) and Stream Control Transmission Protocol (SCTP) are used in this study. DCCP is specially designed to avoid congestion in the network. DCCP is suitable for in time delivery of data and also for its congestion control mechanism. DCCP is an unreliable transport layer protocol, as the real time applications demands for in time delivery rather than reliability. SCTP is another transport layer protocol that provides reliable data transfer. In this research work performance of SCTP, DCCP and UDP has been evaluated using MPEG-4 video over Mobile WiMAX/IEEE 802.16e. The performance of these three transport layer protocols is analyzed in terms of performance metrics like packet loss, jitter, delay and throughput. By analyzing these performance measures it is found that the performance of DCCP and SCTP is much better as compared to UDP but DCCP gives much better performance then SCTP when compared in terms of throughput and packet loss. Comparing SCTP and DCCP with UDP in terms of delay and jitter shows that UDP has less delay and jitter as compared to SCTP and DCCP, but because of less throughput and large number of packet loss, UDP can badly degrade the video quality. So, it is found that the DCCP is the most suitable transport layer protocol for transportation of MPEG-4 traffic over Mobile WiMAX/IEEE 802.16e.