Context: Automated analysis of code review comments (CRCs) can aid in highlighting frequently discussed issues by reviewers from large repositories. Topic modeling is a promising approach to analyzing large natural language repositories. However, CRCs contain natural language text and code references; thus, data pre-processing and topic modeling approaches must be carefully selected.

Objective: This work aims to discuss the various decisions taken and considerations involved in the analysis of CRCs.

Method: We utilized 5,560 CRCs from an open-source system to study the decisions and considerations faced during the analysis of CRCs using topic modeling, followed by an evaluation of the interpretability of identified themes by a domain expert.

Results: We report several observations and challenges in improving the quality of the identified themes, including choices regarding the pre-processing, topic modeling parameters, embedding model, and objective measures of coherence used, which impact the subjective interpretability of the identified themes.

Conclusions: This work offers unique considerations, and the impact of these decisions can facilitate future studies in conducting topic modeling-based analyses of CRCs. Future studies can utilize the technical demonstrator to explore the interpretability of the topics generated from CRCs.