In order to be competitive against the car or the airplane, you need an infrastructure which is maintained for Track Class 5 (or more). However, given that the freight railroads are content with the 60 mph (96 km/h)
limit of Track Class 4 (or even less than that) for their own freight trains, they have no interest in upgrading their tracks (and alignments!) to allow passengers to exceed 80 mph (129 km/h), unless they get fully reimbursed for that expense.
Unfortunately, even if trains operated three times per day (rather than: per week) outside the Q-W Corridor, this would still be a massive waste of taxpayer money. Once you acknowledge the Economics of operating passenger trains, it becomes much easier to understand why VIA invests its limited resources where most Canadians actually live…