Agile software development is gaining popularity, with many organizations across several industries migrating to agile methodologies. Employees are the assets of any organization and it is of utmost importance for the organization’s success to make the best use of available human resources. For this, measuring job performance through performance evaluation becomes crucial. This study aims at investigating the limitations of annual performance appraisal when applied to agile work set-up. When the shortcomings in the current appraisal process are identified, corrective measures for eliminating such gaps are listed out as an outcome of this research study with the aim of making the appraisal system more effective. Survey and interviews are primarily used for data collection. Several important observations are noted as a result of our study. The research findings briefly highlight the good aspects of current appraisal system in the agile work set-up. However, the main focus is to illustrate the limitations in annual performance appraisal with evidence. Consequently, proposals for a better appraisal system in agile work set-up are suggested. Some of them include: need for team performance evaluation, informal and less time-consuming appraisal process, timely and anonymous feedback from all stakeholders.