The standard answer would be : Move on when you have removed all the 1k larger scratch marks and replaced them with finer 4k scratches..
You would need a jewellers loop or microscope to see .

Personally I just do quite a bit around the 4k area and then considerably less there after.

Hope this helps.