Kingston can be a pleasant town to walk around in the late spring. Lots of old [for Ontario] limestone buildings.
London is decent for a budget golfing holiday but the 20+ courses are outside city limits; so you'd probably be renting a car for that. There are abundant riverside bike trails as well but few bike rentals so you'd want to take a bike. In both cases VIA isn't an optimal way to visit London.
Napanee was interesting to walk around for a weekend; it's not really setup for tourists though (restaurants, etc.).
I've not been to Belleville and Cornwall but both seem to attract a few tourists making short trips in the summer.
GO + e-bike [with spare batteries] adds more options for a 3-day weekend. Guelph->Blyth, Uxbridge->Haliburton, Barrie->Penetanguishene, Niagara Falls->Fort Erie, Hamilton->Port Dover [Long Point]