The authors develop an advanced hand motion recognition system for virtual reality applications using a well defined stochastic mathematical approach. Hand gesture is a natural way of interaction with a computer by interpreting the primitive characteristics of gesture movement to the system. This concerns three basic issues: (1) there is no physical contact between the user and the system, (2) the rotation of the hand gesture can be determined by the geometric features, and (3) the model parameter must be optimized to improve measurement of performance. A comparative analysis of other classification techniques used in hand gesture recognition is carried out on the proposed work hybrid with the bio-inspired metaheuristic approach, namely the cuckoo search algorithm, for reducing the complex trajectory in the hidden Markov model (HMM) model. An experimental result is as to how to validate the HMM model, based on the cost value of the optimizer, in order to improve the performance measures of the system. © Springer International Publishing AG, part of Springer Nature 2018.