My guess is that you do like I do: that is, you push that trigger over instead of straight back. When the striker falls, that pressure you put on the trigger allows your shot to snap left (when firing right handed) juuuuuust enough to throw your shot. The further back you go, the more pronounced the shot pushes left.
When I would dry fire at a spot on the wall of my living room, I could see the front sight jump just a tad to the left when the striker fell.....and I then knew it is my trigger finger pushing left. When I really focus on that trigger press, the front sight won't move at all. I have to sort of "pull" the trigger to the right as I am pressing back AND I have to keep my trigger finger from the knuckle to that first joint pointed straight ahead. I've noticed that if that part of the finger isn't pointed at the target, my shot will go left at the 25 yard line.
Really hoping that makes sense to you.
Dot torture drills are also a tremendous help to keep me from pushing left. I can usually shoot a clean dot torture at the 5 yard line IF I do this trigger press method. If I don't, I'll throw one left and since I am right up on the target, it's instant feedback to my foul ups.
My guess is it is your finger/press. You trip that striker and during the fall, your finger pushes the gun to the side. We see it a whole lot at the PD I work at.....so much so that 90 percent of the troops think the M&P just "shoots left" and nobody works on it. When we went to Glocks, it helped but we still have folks throw shots.
That Avery trigger stripe drill mentioned above might be a big help too.