Personally, after 6 years in the St. Lawrence Market area, I can't envisioning moving elsewhere. My wife and I would like a bigger place, but we both agree we wouldn't leave the neighbourhood. Anywhere in the King East to Parliament to Esplanade to Yonge box is great, really, and maybe even bit further east than that.
The best thing to me is that it's such a walkable neighbourhood: 3 major grocery chains, the SLM itself, financial district, every bank, the Eaton Centre (once in a while), Dundas Square, Rainbow and AMC theatres, Harbourfront, the Distillery, a zillion restaurants and pubs... are all within a reasonable walk. Plus the King streetcar passes through if you need transit.
Another thing I like is the mix: on King you have furniture stores that sell tables worth half as much as my condo. A block south there are co-op apartments with many low-income tenants and mom-and-pop restaurants with $5 dinner specials. People are from all different backgrounds, ages, income levels, countries, etc, yet there is still a strong sense of neighbourhood with community events, community gardens, etc. While there is still a bit of scruffiness there's not a major visible homeless or panhandling problem. We have a few local street characters but I recognize them in their normal spots, and they are not aggressive like I've experienced other parts of the city.
There are also some beautiful parks (St. James, Berczy, Crombie) and great historical architecture (Distillery, King East, St. James Cathedral, Flatiron, Front and Church, etc.) Finally, the fact that the waterfront and West Donlands developments are now underway will only make this area more vibrant and central in the future. If the DRL ever becomes a reality it would be perfect.