Lots of good points raised in this thread. My perspective may not carry the day, but I'm just hoping to start a dialogue about the holding pattern for this type of public asset.
I did want to defend this humble section of McCaul Street, though, as its near and dear to my heart:
The Silverstein's site is basically directly across the street from vibrant Baldwin Street Village (ok, its immediately across from the fugly, faux victorians that wrap around onto Baldwin Street, but close enough). Just south of the village across from Elm is McCaul's own commercial strip including Sid's Deli, the now-defunct-but-lovely Sage Deli, Krispy Cream, Cheese smokes and more. Directly south of the site is a 13 storey 60s slab apartment now housing a community living centre. South of that is the eponymous, St. Patrick's Church. A stone's throw away is Dundas and McCaul which is obviously a hugely important intersection for OCAD, the AGO and the City at large. To the north of the faux victorian fuglies is a block of charming real victorians, bay and gable houses and Holy Word Church, followed by a number of office, educational and medical buildings, including repurposed brick and beam buildings and Ronald McDonald House. On the east side, directly north of Silverstein's is, yes, two small hospital parking lots, but these are broken up by a Victorian survivor and then a large parkette to the north and then UofT's great Old Admin building. This is just all in the short hop on McCaul between Dundas and College.
It really is a Jacobsian "complete street", with a multitude of uses, shapes, sizes and architectural styles (even if it's not Toronto's comeliest street).
If inner city surface parking lots really are a necessary evil to ensure proper longterm health care planning (of which I remain to be convinced), I'd really like to see other sites explored first.