Defects in requirement specifications can have severe consequences during the software development life cycle. Some of them may result in poor product quality and/or time and budget overrun due to incorrect or missing quality characteristics, such as security. This characteristic requires special attention in web applications because they have become a target for manipulating sensible data. Several concerns make security difficult to deal with. For instance, security requirements are often misunderstood and improperly specified due to lack of security expertise and emphasis on security during early stages of software development. This often leads to unspecified or ill-defined security-related aspects. These concerns become even more challenging in agile contexts, where lightweight documentation is typically produced. To tackle this problem, we designed an approach for reviewing security-related aspects in agile requirements specifications of web applications. Our proposal considers user stories and security specifications as inputs and relates those user stories to security properties via natural language processing. Based on the related security properties, our approach identifies high-level security requirements from the Open Web Application Security Project (OWASP) to be verified and generates a reading technique to support reviewers in detecting defects. We evaluate our approach via three experimental trials conducted with 56 novice software engineers, measuring effectiveness, efficiency, usefulness and ease of use. We compare our approach against using: (1) the OWASP high-level security requirements and (2) a perspective-based approach as proposed in contemporary state of the art. The results strengthen our confidence that using our approach has a positive impact (with large effect size) on the performance of inspectors in terms of effectiveness and efficiency. © 2020, Springer-Verlag London Ltd., part of Springer Nature.