In this paper, wave propagation in a hollow waveguide with a graded dielectric layer is studied. Analytic formulas are derived for the electric field components as well as general analytical results for the reflection and transmission coefficients for propagating waves. These results are all valid for waveguides of arbitrary cross sections, and the derived reflection and transmission coefficients are in exact asymptotic agreement with those obtained for a wry thin homogeneous dielectric layer using cascading and mode-matching techniques. Furthermore, the power transmission, reflection, and absorption coefficients, as functions of frequency and layer width, are studied, showing the expected behavior of these parameters. The method proposed in this paper gives directly applicable results that do not require cascading and mode matching, while at the same time having the ability to model smooth transitions that are more realistic in several applications. (C) 2020 Optical Society of America