You are not using enough grit. I usually use a 325, 400 or 140 if it really is out of flat. But as said, it does not need to be perfectly flat, it is up to you.

A piece of 220 or 320 wet and dry...