I agree that the Google we see today is a far cry from what it once was, and as a result of this, we should be wary in regards to what powers we give them. This goes for all tech giants.
Personal data is the most valuable commodity in the modern world, and Google has an incredible amount of power in terms of how that data is used or sold. The collection, retention, and sale of personal data is largely unregulated in most countries, and it is important to remember this. A tech company like Google only needs to cover itself by outlining its intentions in a set of terms and conditions, which are required to be accepted before using any of their products or services, which nobody reads anyway. This pretty much gives them carte blanche in regards to data collection, as long as they have it in the terms and conditions.
There were a lot of different methods of data collection proposed in Sidewalk's report on the subject, called the "Digital Innovation Appendix". These mainly involve the collection of aggregate data, mainly for the optimization of the services they planned to offer within the community (trash collection, adaptive signals/lighting/heating, etc), allowing for adjustments based on traffic loads at certain times of day. This is fine. However, you can actually go through the list and start to see the implications of the type of infrastructure and modelling they intended on doing. Some systems, namely the "Outcome-Based Building Code Monitoring" infrastructure have a lot of worrying capabilities, such as indoor sound monitoring equipment (outlined to "ensure tenants stay below acceptable noise thresholds"), indoor air quality monitoring equipment, and more, such as vaguely defined "tenant diversity tracking", combining the data in a BIM-type setting. Things like this, combined with other metadata that Google would most certainly be collecting from its residents, could very much allow for a "social credit" type system emerging within the community.
Sidewalk does state that they aimed to collect the least amount of personal data possible, but after approval, a simple alteration to the terms and conditions could have changed that. The issue here is not about the Google of today, but what the Google of tomorrow wants. The concerns are about the possible extent of what could have been collected, not necessarily what Google had "promised" to collect.