It seems to me that hardness would have more to do with the mineral composition of a stone than the density.

Maybe I'm not thinking of hardness the right way though.