It is true that there is a significant population east of Quebec City.
It is also true that this population is not arranged into the sorts of population centres you could efficiently serve with a high speed rail corridor between Quebec City and Halifax. We're talking about building 800 kilometres of high-speed track to serve Halifax, Moncton, and maybe Truro. (It may be tempting to put a station somewhere like Edmonston or Grand Falls, but these places don't have nearly the population to support it. These stops could even produce net-negative ridership: they would slow the service down and make tickets more expensive, deterring riders between other stations, while adding very few riders themselves.)
Whereas Southwestern Ontario, with just 400 kilometres of track, you can have a pick of two alignments, both of which thread together some decently large population centres. (Toronto -> Kitchener -> London -> Windsor, or Toronto -> Hamilton -> Brantford -> London -> Windsor.)