Victoria Park is half way through the route. Many routes will have only one or two people on them for the latter half of the route. The 510 is often empty or near to it crossing Lakeshore at night. The further from the subway station the less desirable the mode of transportation. I have a hard time believing the 42 bus is driver only at 7pm.
If you cut back the service times of a route a couple of hours you are going to see reductions in the number of riders in the hours prior to the new closing period. Obviously the routes that run on the primary arterial routes in the city are going to have greater demand because there is greater employment and nearby density on those routes. However, if the 42 bus truly is empty at 7pm with its 7000+ daily riders and a similar cost recovery ratio to many non-arterial routes in the city the removal of that route would mean the removal of one heck of a lot of routes.
For many of the groups arguing for transit stops out their front door the additional walk is tough to manage... and I am pretty sure, except for at Finch Station, Cummer stops are much further from Finch or Steeles than 200m, and many riders who have a Cummer bus stop as their closest probably walk 200m to get there.
If it were a luxury people would park their car which sits unused most of the day for the luxury of the bus.
There is also the people displace by bus cancellations to think about. They took the bus, despite its inconveniences, and now the route they take is getting cancelled. For some which are unable to walk that distance and don't have the money for an alternative this takes away their mobility, for some who chose to save money because the stop was conveniently located they might end up paying more to drive, and for some they will walk further and have a longer trip time to go onto another bus route which is already crammed and possibly not find space. Only for those who both cannot afford the alternative nor walk the distance is transit truly essential. The power outage a couple of years back proved large numbers of TTC users are quite capable of walking from the CBD to Eglinton.