Lyric discussion by trademark 

I've always taken this song to mean that the narrator is tired of life and situations that cause anxiety and he would much rather be with the person he cares for, as in "slip away" from the world and be in a comfort zone. I agree with the autumn sweater represents comfort and security, and the fact he has nothing to say could mean that he's comfortable in the quiet or he has nothing to complain about.

with the idea that the autumn sweater*

An error occured.