I think you are been dramatic here.
Gentrification isn't pushing them out of the city - the city doesn't ends at Sherbourne st on the east, does it? The issue is Sherbourne and Queen is take an unfairly high share of social housing/poverty, which is not good for the neighbourhood, nor for the city as a whole, nor for the poor people. If the poor always see nothing but poverty and drug abuse surrounding them, there is no hope for them. This is why such social problems should be spread out a bit. You don't want to have 6 homeless shelters congregating on 0.6 sk miles. That doesn't solve the problem, if not exacerbate it.
I don't think "various social services" are exclusive to downtown, and downtown is definitely not the only place that is livable without a car. Maybe social services should be spread more evenly too, especially to some of the current better neighbourhood, such as the Annex, Yorkville, Rosedale, Casa Loma, Leslieville, King West, Deer Park, Yonge/Eg, etc, so that they all have their fair share of social housing? I certainly won't consider that pushing out the poor. What makes it so important for downtown east to have such a high concentration? Toronto isn't just downtown and the outer suburbs. There are many areas in between. In the end, no particularly neighbourhood in Toronto deserves a reputation for being the hotbed of poverty or crime like Moss Park.
We all advocate mixed income neighbourhood, which I agree, but is Moss Park really mix income?