Definitely. And those 20 minutes are the best case scenario. Take a look at train 647: it's scheduled to take 50 minutes to cover 16 km from Fallowfield to Ottawa, which is an average of 19 km/h:
View attachment 505318
This additional time is due to the poor track configuration between Fallowfield and Ottawa stations. The segment is a single-tracked, with a single short siding near the Rideau river, and sidings at Ottawa and Fallowfield stations.
But although Fallowfield has a siding, it does not have a proper second platform. So when trains use the siding, passengers need to walk across the main track to get to the building. Which obviously makes the main track in accessible to trains during that time, and makes it impossible for two trains to serve the station at the same time.
Here's how the meet works between trains 644 and 647:
At 17:00, train 644 arrives and serves the station. I don't know which track it uses, but let's say it uses the siding.
View attachment 505341
At 17:03 it departs, but the single track ahead is occupied by train 647 coming westbound. So it can only roll forward to the end of the siding.
View attachment 505339
At 17:06 train 647 arrives at the station on the main track which is now available because train 644 has finished loading.
View attachment 505338
Finally once the switch is thrown, train 644 can proceed eastward.
View attachment 505337
As you may have noticed, this manoeuvre only accounts for about a 5 minute delay, which is far less than the 30 extra minutes in train 644's schedule. But this manoeuvre only works if both trains are exactly on time. If train 644 is any more than 3 minutes late, it wouldn't be able to clear the station in time for train 647's arrival. It appears that the other 25 minutes of extra time in 644's schedule are schedule padding which allows VIA to always prioritize train 647 over 644 (which makes sense since delays to 647 could have cascading effects down the line).
So in the very likely event that train 644 is more than 3 minutes late, it wouldn't be able to start unloading until 17:09 when 647 clears the platform. So it wouldn't depart Fallowfield until about 17:13 at the earliest.
This situation could be improved by building a second platform at Fallowfield station, allowing two trains to serve it at the same time without delaying each other. Extending the double track eastward from Fallowfield to the Rideau River would further reduce the delay caused by train meets by providing a larger margin of error for timing the meets. Increasing the track speeds to 60 mph along the current 45 mph zone from the Rideau River to Ottawa station would reduce the amount of time trains need to occupy the single-tracked segment, additionally reducing the delay caused if two trains meet at the single-tracked segment.