If I "stray" with my first shot (regardless of where the gun started . . . . on a table, in a box, from the holster, unloaded, etc), I have determined that 90% of the time it was because I was not gripping the gun firmly enough.
And it wasn't because I was consciously trying to hurry. It was because I got "lazy" with my grip.
I can even go to the range to test ammo or sight in a pistol and be almost assured that the first group or two that I shoot will be worse than subsequent groups. Then I remind myself to grip the damn pistol FIRMLY, and things straighten up.
I have recently started repeating "FIRM GRIP" to myself just before the start signal when competing. Still have a ways to go to get that drilled into my head.