Yeah.. welcome to the casual club. I've had a lot of shaves go supernaturally well when I'm not going slow, not too hasty but I just kind of "let go" and go quick. I think what you're doing is releasing yourself to go deeper into your hand's knowledge, get a little of that expert's touch we see with some of these barbers shaves. It can be surprisingly thorough. Most of my shaves are between 15-18 minutes, I've had a few around 10-12 and they were as good as the others (my first month's average was 40-45 min).

Maybe this mentality works for you. I know my performance can suffer somewhat if I try too hard.