This paper introduces the Gray Wolf Optimized Convolutional Transformer Network, a combined deep learning framework aimed at accurately and efficiently recognizing dynamic hand gestures, especially in American Sign Language (ASL). The model integrates Convolutional Neural Networks (CNNs) for spatial feature extraction, Transformers for temporal sequence modeling, and Grey Wolf Optimization (GWO) for hyperparameter tuning. Extensive experiments were conducted on two benchmark datasets, ASL Alphabet and ASL MNIST to validate the model’s effectiveness in both static and dynamic sign classification. The proposed model achieved superior performance across all key metrics, including a accuracy of 99.40%, F1-score of 99.31%, Matthews Correlation Coefficient (MCC) of 0.988, and Area Under the Curve (AUC) of 0.992, surpassing existing models such as PCA-IGWO, KPCA-IGWO, GWO-CNN, and AEGWO-NET. Real-time gesture detection outputs further demonstrated the model’s robustness in varied environmental conditions and its applicability in assistive communication technologies. Additionally, the integration of GWO not only accelerated convergence but also enhanced generalization by optimally selecting model configurations. The results show that GWO-CTransNet offers a powerful, scalable solution for vision-based sign language recognition systems, combining high accuracy, fast inference, and adaptability in real-world applications.