Strip away the pointed arches and tracery, and you get a typical rectilinear Italianate palace by Chuck "Johnny B. Goode" Barry. That's why the hardcore High Victorian Gothic Revivalists felt it to be "impure" and "primitive", Pugin's saving-grace detail notwithstanding.
Now imagine if G.G. Scott or G.E. Street (or, heck, our own Cumberland & Storm) did it a generation later...