I've done some research on the role of the olympics in helping tourism since I'm doing my masters in tourism planning.
Olympics can have a positive impact on tourism if used properly. In terms of creating a brand, enhancing a product and improving amenities they can go a long way. Sydney, Seoul and Barcelona are strong examples of that. Places like Beijing, Atlanta, Los Angeles and Lillehammer did very little and just expected people to come because they held the Olympics. It's not that simple.
here's a good article describing the difference between Sydney and Beijing's approaches:
http://www.travelwires.com/wp/2009/02/will-2010-tourism-succeed-like-sydney-or-flop-like-beijing/
It's also difficult to measure the impact of the Olympics on tourism for places like Nagano, Sydney, Salt Lake and Athens because of 9/11. Global travel fell around the world. It's also impossible to measure profits and losses with Olympics. Anyone that tries to talk about that stuff has no clue and for me it raises a flag. The intangibles are impossible to measure.
My own theory is that cities that have been off the radar can benefit significantly by putting itself on the map (provided they use the olympics properly). I think Toronto would fall into this category because it is similar to Sydney and Barcelona in size and international presence. However, places like London, Beijing and LA, which already have an established world renown tourism product are likely to see little improvement because they already have an international presence and anyone who wanted to go there probably has (easy access, high frequency of flights, etc) and its doubtful anyone would be seeing the products these cities possess for the first time during the Olympics.
Ultimately, I think it's impossible to tell if an Olympics is a good or bad thing. Every Olympics seems to be different and have a different effect on their city. I think it ultimately comes down to management and taking advantage of the opportunity. If you fail there, you're going to fail to see an after-glow.
Unfortunately, the Pan-Am games won't have an after glow. We'll have a spike in tourism during the games and then it will return to normal. Which is still great. Anytime you have a spike, it means money is being made, and heaven knows our tourism sector could use an infusion of cash. And yes we will have tourists during the games. Even the athletes are tourists.