For those of you that didnt have to lap much off the 4K side of your Norton.....congratulations, you have been fortunate. My Norton needed to have about an eighth of an inch lapped off the 4K before it was smooth enough. Mine had quite a bit of porosity and texture in the outer most layer.
I guess the right answer is "keep lapping until the surface is smooth and free from roughness, and don't be surprised if you have to take off quite a bit".
BTW, the DMT 325 is a really good choice for a lapping stone. I have the Norton lapping stone, a DMT, sandpaper and a Shapton DGLP. The DMT is probably the sweet spot in the group. The Shapton is better, but there are definitely diminishing returns at its price point.
Edit: One more thing, don't forget to put a small radius on the edges of the stone after you lap it.