This song has a lot of depth to the lyrics. "Birds flew backwards" refers to the narrator's perspective as he is moving faster than the birds in the same direction, possibly from a train window or the back seat of a cab. Seeing the birds, presumably swallows, he anticipates the end of winter and beginning of summer. The specific grammar used between the two verses implies a sense of urgency by the end of the song. Saying "won't be long" seems as though a casual observation, then later "can't be long" delivers an impatience that summer hasn't yet arrived.

I imagined this, myself, to be a person who's had a really bad winter (when it rains, it pours, right?), and is looking for any sign to believe that things will improve. Hence the urgency to see summer begin. Taken more simply, it could be a child or teenager sitting in class waiting for summer holiday.