Then you might as well refer to Manhattan at large as "downtown" New York City.
I think this is essentially the problem here. When we're talking about the former City of Toronto, we have no word for it really. People say "Downtown" as a substitute for "the former City of Toronto," which is too much of a mouthful in normal speech. Unlike in New York, we have no separate name for the central city (which arises from the very different histories of New York's formal division into boroughs, and Toronto's division into informal boroughs post-Megacity). Since amalgamation, the entire way we locate ourselves within Toronto has become confusing and imprecise. There's no more easy distinction between "the City" and "Metro." Even terms like "East End." "West End," and "North Toronto" have become ambiguous. Sometimes, for example, the media will say something that happened in Scarborough occurred in the East End (which is sort of understandable if not untraditional). At other times, the media will identify an area of North York as North Toronto (again understandable, if less forgivable). I've even heard (though rarely) Mississauga described as part of the West End.
The other reason, I think, for the confusion has to do with downtowns elsewhere in Ontario. In many towns and cities in this province, downtown is any part of the central city that has streets lined with retail, dense housing, etc. In short, "downtown" to many Ontarians, if not Canadians, means urban and
vice versa. Someone coming from Brampton, Peterborough, Oshawa, etc. might assume any area like this in Toronto is part of downtown (as these parts of the city have "downtown" qualities to them). The Danforth
looks downtown to these people, St. Clair
looks downtownish, etc. even if locals would not identify them as such. Then there are motorists, for whom "driving downtown" means driving anywhere without free parking, with lots of pedestrians, streetcar tracks, etc. (and this includes people from the 416 as much as it does those from the 905 and beyond).
No. The city does not claim this.
The Central Area is Bathurst to the Don, the lake to the CPR tracks and Rosedale Valley.
There is the phenomenon of the incredible shrinking downtown, which isn't by any means a Toronto thing. From my home in the West End, going to College and Spadina (for example) is "going downtown." But going from College and Spadina down to Yonge and Queen, let's say, is also "going downtown."