Focusing on the past 25 years or so, I'd say U of T has had a fair amount of quality new architectural contributions to their downtown campus. The ones that stand out for me are: the Terrence Donnelly Centre, Woodsworth College residences, Bahen Centre, and Max Gluskin House.
On the enigmatic side, but I actually like the Graduate House too at Spadina & Harbord. Personally find it to be one of the more interesting miscellaneous/odd ball buildings in the city.
Trending towards the hit or miss territory for various reasons but I'd say the following have more positive than negative aspects: Rotman expansion, Leslie Dan pharmacy building, Goldring Centre, Jackman Law building, New College III, Daniels faculty of architecture new addition, Myhal Centre.
If expanded to 50 years, I think there's a lot of understated gems especially various buildings along Huron Street between Harbord and Ursula Franklin Street.