Perhaps you are a bit dehydrated in the morning when you shave. Several hours later you have rehydrated, causing your skin to plump out and whiskers to retract a bit more.