“I may not agree with what you have to say, but I will fight to the death for your right to say it.” – Voltaire
This principle of individual liberty is the foundation of Western Liberal philosophy. If we expect others to leave us alone to do as we please with our property, then we must leave others alone to do as they please. Even if we wouldn’t choose the same. We can talk with them, reason with them, bribe them, try to persuade and convince them, but in the end, it’s the owner of a thing who gets the final say in what happens with a thing.
It is a wonderful service for the neighborhood to schedule a day for us to get together and talk about this issue and how it affects us, but ultimately no one has a higher claim to this property than the owner. If they decide to rebuild, then that is their right as the owner. No one can charge them money or stop them from building.
To claim otherwise would be to say that someone besides the owner has a higher claim to their property than they do. That doesn’t make sense in a post-enlightenment society that claims to respect the rights of the smallest minority: the individual.