Thanks so much, Mustapha, for the book scans! I could only find 2 mentions of our Ansley Castle in Toronto, two photos from June 9th 1923 a couple of years before it was torn down, at
http://collectionscanada.gc.ca but the records don't have the scanned photos available online, unfortunately.
The book Opportunity road: Yonge Street 1860-1939 By F. R. Berchem mentions Yellow House, Ansley Castle, Glen Castle, and King Dodds racetrack there. See page 94 and 95 at
http://books.google.ca/books?id=KEe20HX9qdMC&lpg=PA94&ots=-VFbarXVBz&dq=%2B%22Ansley%20Castle%22%20-england&pg=PA95#v=onepage&q=+%22Ansley%20Castle%22%20-england&f=false
The two images then of Ansley Castle have squared sided turrets, unlike the image I originally posted... Here's the images we have, with the unknown turret in the back
Still a mystery then!