You are missing a few things. Take this example:
A dump truck coming from the east end of the city needs to go to a Bathurst/Dundas construction site. Right now they would take the DVP down to the Gardiner and get off at Spadina. With the Gardiner out, the boulevard is no longer the fastest route, so now they might get off at Bloor or Richmond and tear through downtown in the middle of the night.
As for the pedestrian argument, the remove option won't get rid of the similarly nasty Lake Shore crossings at Spadina/Bay, etc. under the Gardiner.
A wide boulevard wouldn't be perfect either, as to accommodate the traffic the signal timings will probably give a short north/south duration, resulting in 2-stage pedestrian crossings, similar to University Ave.