Lyric discussion by gsokolow 

A bunch of years ago, 1970's?, I went to a small concert with the folksinger Chuck Mitchell (Joni Mitchell's first husband - and he hated to be refered to as that). Well he gave a great concert to less than 100 people at the "Down Under" coffee house at FSU (Tallahassee Florida). He started playing the chords to a song and told a story about he and Bob Dylan were up in the attic of someone's house. He told us that Dylan actually shared the meaning to him. He said it referred to the one common musician in all New Orleans Jazz funerals. Wealthy people would get a marching band in fancy uniforms, poorer people might only get a few musicians. He claimed the tradition was there was always a Tambourine player who didn't march in formation, or where a matching uniform. This was for rich people and poor people. He claimed that the Tambourine Man symbolized DEATH and that's it. The entire song is about death and not drugs. He then did a great version of the song with all the verses. The death interpretation makes perfect sense if you listen to all the verses.
especially: Take me on a trip upon your magic swirlin' ship, My senses have been stripped, my hands can't feel to grip, My toes too numb to step, wait only for my boot heels To be wanderin'. I'm ready to go anywhere, I'm ready for to fade Into my own parade, cast your dancing spell my way, I promise to go under it.

When you are dead all your senses are stripped.

Chuck Mitchell's story sounded very convincing, and I could not have come up with this interpretation myself.

i didn't know all that stuff about the tambourine player, but I'm so glad I'm not crazy thinking its about death!

@gsokolow Gerat stuff. Thank you for this information.

An error occured.