Well, let's attempt to find the meaning of this.

"So the siren went off
It chased you out of your mind"

-- I'm thinking something big happened. Some guy had a red alert for something set in the back of his mind, and it (more or less) 'went off'. The chasing part seems to indicate some struggle. You don't have to chase something unless it's getting away from you. So, he's trying to run from whatever caused the 'siren', and it catches up. Drives him crazy.

"Just when I started to grow a garden"

-- I see people using gardens as metaphors for feelings or relationships. Maybe this is like, "Just when I started to find friends/love".

"How selfish of me
But I need something to build on to
You can't leave me here"

-- The selfish part could be sarcastic. If her making a new relationship startled the guy so much as to put off a 'siren' in his head, then she's saying, "Oh, sorry for hurting you back there when I was just trying to be happy." "But I need something to build on to" - She's explaining her reasoning. If she doesn't get that love out of her 'garden', what else can she do? The guy is coming and going, she can't just be by herself.

Thus, the metaphorical castle comes in.

"Alone in our castle
Alone in everything we have
I'll never see your face
The way it is in our castle "

A stronghold, a place he built perhaps for her - he's gone and she's just sitting there with all their belongings. The chorus can be taken two ways. Particularly the lines, "I'll never see your face / the way it is / in our castle." It could mean, "I won't see you the same way if I'm outside/no longer protected", or "I won't see you again, because of this castle (because that's the 'way it is'."

"So I kidnapped your sweet sparrow
So I am dancing around on your pain"

I took something precious of yours, and I'm spending my time enjoying how hurt you are. May not be the most moralistic thing to do, but this person obviously has problems with their counterpart. Of course, you can tell in their voice a part of them isn't really enjoying it.

"The hallways echo your longing
We both know what needs to be done
But something is holding on"

The other person wants what they lost back. The sparrow? The singer knows they should give it back. But a part of everything won't let her.

"Alone in our castle
Alone in everything we have
I'll never feel your skin
The way I do in our castle "

Same as the last chorus. Though this seems to indicate, "You won't feel the same if you aren't here," as opposed to "I can't feel you again, because you're gone." So maybe they both mean that? If it isn't the castle, it isn't the same. They aren't the same person.

"Love oh love
I want to believe in love
I want to believe this feeling
That I feel deeply for you"

Maybe their relationship with each other has been falling apart. And she, the singer, wants to keep it together.

"But if you can't
Turn away and go
Don't turn back
And I'll do the same "

Kind of self-explanatory. If you don't want to, I guess I'll have to agree with that. Just walk away and deny everything. Repeat choruses.

So, perhaps, the song is about two lovers falling apart. She finally found happiness, which she couldn't get, maybe because of the castle's protection, but he was against it. Regardless, she needed that little bit of risk, but he doesn't care. He leaves her there. She's forever in the walls of it, of these barriers.

Given that this was written after the manga series NANA began, I'm not saying this is written for it (though she did perform a lot for it), but it could fit very well with a situation. The relationship of two of it's characters - Layla and Takumi - the song is precisely about, in that context.

He spent most of their lives building this 'castle', to protect her from the real world, and whenever she tried to build relationships outside of them he'd disapprove. Whether out of jealousy or concern, maybe both, she was effectively locked into a cage.

The sparrow part may even talk about herself: "I kept my singing from you, which is the thing you cherish more than anything else." She knows she has to return, to recordings and things, because they're all part of the same band, but her broken heart keeps her. If she stays in the castle, or the band, she'll be near him, as a member and vocalist, but she'll never get what she truly wants: to be WITH him. Their castle prevents it. All she can do is sit and reflect on what they've gained - the fame and money - and his love, what she'll never obtain, alone in their castle.

Finally, the bridge, she's well aware she's in love with him but she won't outwardly show it if he won't reciprocate. Most of the time. It's really fitting for their latest situation, actually - he -does- reciprocate, but only for a moment. He has to leave her and pretend nothing happened, as does she, because they have reputations to upkeep (her as the 'singing princess' of their band and him as a respectable husband to the media).

Anyway. Such a great song.

Well, apparently the guy she liked started to have a nervous breakdown or go crazy or something; yet that was when she was finally starting to build a life for herself. She felt guilty, but she had to think what was best for herself and she was desperate to finally have a life for herself; so she left him.

Thus leaving him "alone in our castle"; the little sanctuary of isolation from the world the two of them had formed; before she found a life of her own.

Yet she still misses him after she left him;
"I'll never feel your skin the way I did in our castle".

Its a sad song about two broken people; one of them had a chance to repair, and the other didn't. So tearfully, in guilt, she left him so that she could get one last chance at a life; thus they split off in two opposite directions.

...but she still, you know, misses him.