David Audretsch has made significant contributions to our understanding of the role of knowledge spillovers for innovations and growth. This paper follows this line of research in examining the link between education, human capital spillovers and productivity. Human capital spillovers arise when the presence of individuals with high levels of human capital makes other workers more productive. If higher education is associated with human capital spillovers, a social return to education is generated. We use firm-level production functions to estimate the social returns to higher education in Sweden. The data include more than 50,000 Swedish firms and cover the period from 2001 to 2010. This was a period when Sweden experienced a rapid regional expansion of higher education, and the share of Swedish workforce with higher education has increased dramatically over the past decades. We find economically significant spillover effects from highly educated workers and that a 1 % increase in the share of educated workers is associated with a 0.4-1.0% increase in productivity. When controlling for university-based R&D and business services, the spillover effect is significantly reduced. We also find an economically significant decline in the spillover effect over the 10 year period. According to our estimations, the spillover is positive at the beginning of the period and gradually diminishes by the end of the period, such that we no longer find any significant spillover effect. We interpret this as marginally diminishing social returns to education. The results have policy implications for higher education (We gratefully acknowledge financial support from the Marianne and Marcus Wallenberg foundation and the Kamprad family foundation. We also grateful for the valuable comments provided by Pontus Braunerhjelm, Gunnar Eliasson and Hulya Ulku. Peter S. Karlsson has provided valuable statistical assistance). © Springer Nature Switzerland AG 2019.