It has been suggested that physical or social activity is associated with fewer sleep disturbances among elderly people. Women report more sleep disturbances than men, which could indicate a variation in activity patterns between the genders. The aim of this study was to investigate associations between sleep disturbances and leisure activities in men and women (n = 945) aged ≥60 years in a Swedish population. Sleep disturbances were measured using eight dichotomous questions and seventeen variables, covering a wide range of leisure activities. Few leisure activities were found to be associated with sleep disturbances and their importance decreased when the models were adjusted for confounders and gender interactions. After clustering the leisure activities and investigating individual activities, socio-intellectual activities were shown to be significant for sleep. However, following adjustment for confounders and gender interactions, home maintenance was the only activity significant for sleep. Being a female increased the effect of home maintenance. Besides those leisure activities, poor/fair self-rated health (OR 7.50, CI: 4.27-11.81) and being female (OR 4.86, CI: 2.75-8.61) were found to have the highest association with poor sleep. Leisure activities pursued by elderly people should focus on activities of a socio-intellectual nature, especially among women, to promote sleep.