I expect most Rosedale taxpayers would fight tooth and nail any move to introduce people of the lowest economic rungs into their neighbourhood. They carry considerable clout and expect their politicians to back them up. Nimbyism at its finest.
And we often assume displacement is to a more remote area because that's what has traditionally happened... the poor get pushed out to the margins, where services are leaner, rarer; where transit options are fewer. This happens to any gentrifying neighbourhood in Toronto, by the way. If you were living in Corktown and found you could no longer afford to live there, you're hardly likely to move even closer to the city core - especially today, with skyrocketing property values and commensurate rental rates. No, you'd probably be moving further away from the core, to places where services and amenities tend to be far more dispersed. If things are to change, the old reflexive measures have to be re-evaluated and revised; the city as a whole must adjust. Such change tends to come about quite slowly.
By the way, Leslieville will be playing host to a homeless shelter soon, right on Leslie itself. But I'm pretty sure it's by definition a temporary shelter, not permanent housing.