Did you lap on sand paper?

As has already been said it could be bits of grit becoming embeded from that.

I lapped my Norton 4k/8k yesterday with sand paper and encountered the same problem....