I couldn't disagree more. When I lived in the area (for approx a year) I smelled the sugar factory maybe once every month. Some people loved it, I was indifferent to it, but it definitely was not "ever-present". Maybe they've changed the factory, but I would walk by on a semi-daily basis and rarely would smell it. The lake can smell sometimes, but that was rare too and the price you pay for being near a body of water. Besides, it's not like the rest of the city smells like roses - every area of a city has strange smells from time-to-time. I was just walking around King and Church this afternoon and got a huge whiff of the knock-you-off-your-feet smell of sewage for virtually the whole block.
Fact is, this area is located dead centre along the waterfront of one of the world's major cities, steps to the city's financial district, transportation hub, and entertainment complex (Scotiabank Arena) and a short walk to some of the city's major cultural spots (St Lawrence Market, Harbourfront Centre, Toronto Island). That's a pretty damn impressive list of reasons to live there now. In 3-5 years when the blocks around this that are currently under development become developed, and the area becomes dense with parks and new supertall skyscrapers, it's going to be hard to find a more desirable place in the city to live, in my opinion.