Accurate classification of underwater boulders is essential for a range of marine applications. Manual classificationmapproaches are labor-intensive, prone to subjectivity, and unsuitable for large-scale survey applications. In this study, we introduce a new benchmark dataset, SonarBoulder, constructed from high-resolution sonar imagery collected off the coast of Sweden. The dataset includes both balanced and imbalanced versions to evaluate model performance. We conducted a comparative analysis of seventeen classification models, including traditional machine learning models (e.g., Random Forest, KNN, SVM, Logistic Regression, AdaBoost, Naive Bayes, ANN, and XGBoost), deep learning methods (e.g., CNN, VGG16, ResNet50, ResNet101, EfficientNetB0, and InceptionV3), and transformer-based frameworks (e.g., DeepViT and ViT). On the balanced SonarBoulder dataset, the deep learning model ResNet101 achieved the best overall performance with 97.75% accuracy, 97.92% precision, and 97.66% F1 score, while the transformer model DeepViT achieved the highest recall at 98.59%. In the imbalanced scenario, InceptionV3 achieved the highest precision (97.26%), recall (95.83%), and F1 score (96.54%) as compared to the other methods. Our results demonstrate the effectiveness of using deep learning and transformer-based models for underwater boulder classification.