Any ideas why my .303 British Enfield bolt action is not holding a pattern?

Here is the info:
Last year tried sighting it in and it failed miserably. I was sighting in at 50 yards on a shooting bench on a sled. Adjusting the crosshairs on my Nikon scope did not correlate to getting closer to the bullseye. Once I kept the clip in the rifle, it got better, but still no really recognizable pattern.
Got frustrated, thought, "maybe it is the operator." Grabbed my 30-06 and was putting bullet holes touching each other...

So, in order to figure it out, had my Brother-in-law switch the scopes on my two rifles (he has the tools to do it properly-I do not).

This time at the range, same system: 50 yards, little wind, bench and sled. 30-06 is hitting within 4 MOA right where is should.
.303 responded to scope adjustments, but no discernable pattern within 6 inches of the bullseye (I believe that makes it 12 MOA).
The barrel is free floating.
The only idea I have yet to try is switching ammo.
Any other ideas?