Actually, contrary to "monied", the Annex bottomed out in the 50s/60s--many of the then-unfashionable late Victorian mansions had become white elephant rooming houses, or redlined for apartment redevelopment along St George/Spadina/Walmer in particular. Not all that different from what was happening in Parkdale around that same period.
However, one thing that the Annex had which Parkdale didn't was a "cultural class", thanks to its proximity to U of T, among other things--many of those inhabiting said old hulks and rooming houses happened to be students, professors, academics, intelligentsia, etc. Monied or not, they were the "cultural elites"--and that's what gave the Stop-The-Spadina movement its "push"...