When we arrived in Toronto in July, 1979 I was working from 55 University - I do not recall seeing the CP Express Freight building either in place, or being demolished. I have always walked around during my lunch breaks, looking at what was happening - I definitely do not remember either the building, or seeing it being demolished - even though it would have been a short block away.
On a side note - there was a very similar and comparable railway express freight forwarding building right in downtown Montreal, on the west side of Rue Peel, between Rue St Jacques and Rue Notre Dame. From what I recall, it was still in place when we left Montreal in 1979 - it however is also long gone as well.