There's also a lot of suspicion around photographers because there's a fear that someone (the Ministry, building inspectors, a union, competing contractors, neighbourhood associations, etc...) is conducting surveillance to try to catch someone in a "Gotcha!" moment. It doesn't matter how carefully you try to comply with the rules and regulations, over the course of a day or a week everyone will slip up and have their safety glasses off for a minute, or be a couple minutes late back from lunch, or whatever other infraction someone might commit, and then that photo gets dragged out as evidence. I always hated having my photo taken, and even when I needed to take site photos I was always careful not to include any people in anything.
I'm not saying the guy was right for coming out and yelling about privacy. He was in the wrong, and he can be photographed in public. But it might be helpful for everyone to have some perspective and empathy for why he's saying it.