Lyric discussion by PrenticeGT 

Solitaire is a losing game. The odds of winning are not very good. So when he says that's all he was playing, it's a metaphor for him being a loser basically.

An error occured.