I do pretty much the same thing, but I wash off the "weird smudges" with a papertowel and dish soap. After the red rouge the finish is kind of cloudy. Then it's maas an a felt wheel. Wash it again...