I would suggest Canoe, at the TD center (54th floor of TD Bank Tower). You definately need to make a reservation, especially if you want a window seat.
I've been there once before with a good friend, and the place rocked. Mind you, we were there for lunch, which was cheaper, but the service was exellent, and we left a great tip. We were treated wonderfully, even though we were not wearing suits and ties- I was dressed in a kimono and white hakama, and my friend was just wearing his everyday clothes.
Since the place caters primarily to the "suits", I don't think you will have many problems with screaming babies, unruly kids running around, and so on.
You don't have to do the whole heavy dinner thing- you could perhaps get appetizers and desserts, or just go to the bar for drinks, stuff like that. And you can preview the hours, menu, etc, here:
www.canoerestaurant.com/