There's only one taxpayer. It's not a discount, it's a shell game. You (as a citizen) own all three levels of government.
The problem with the Olympics is you get a very bad return on your investment - by any measure. Say we spend $5B on Games infrastructure, $10B on general infrastructure and $5B to run the thing. $2B of that Games spend is for a stadium we don't need, and the remaining $3B is spent on overbuilding venues and paying a premium to accelerate construction. The $5B in security and logistics is just money flushed down the toilet. The $10B in true infrastructure spend is the real legacy, but if it's that important we should be building it anyways. Wrapping it in the Games is just a hack move by visionless politicians who lack the ability to sell a plan based on its own merits.
In contrast, we could spend $1B to improve our regional sports & recreation infrastructure and get far more bang for the buck. A few hundred million of the $5B in event spending (for what one assumes would actually be a pretty decent party) could fund a shitload of Pride Parades, Caribanas, NXNE and who knows what other events that would actually bring tourist dollars into the city. Olympics don't result in tourist booms before or after the games and spending for the two weeks is partially offset by all the Torontonians who would get out of dodge for the two weeks. Meanwhile, the $10B in basic infrastructure (DRL etc...) should be spent regardless of whether we get to run around with a flame on a stick.
So basically we'd be wasting about $5-10B by hosting the Olympics in order to secure $10B in infrastructure spending we should be investing regardless. Not to mention the time and effort and agenda-hijacking for seven years at all three levels of government. That's why someone who is serious about city building should be against the Olympics, not for it.