What if society deemed driving a car with small children in it were immoral?

What if living in a house made from trees were immoral?
Would you accept that, just as society frowns upon smoking or drinking or wearing fur?

Or is there maybe a deeper drive for good morals despite your beliefs?

There's tons of things deemed immoral in society that aren't bad for your health or others. Yet the constants, such as stuff we know are harmful, we have very strong belief in, whereas the other ones, deemed immoral such as it is by society, we care less about or don't follow at all.