Yeah, it's usually hair or dirt. Are you going from a lower grit stone to a higher without rinsing the razor? I assume you're rinsing, but that would explain it too.