I don't understand why there should be a problem using higher capacity mag. G26 specified the option of using higher capacity mag. I can't say anything about aftermarket mags, just use Glock mags.
You sure it's not the ammo you use? Do you have any problem using the original 10 rounds mag that came with the gun? If you have the same problem with the original 10 rounds mag, then it's not the G17 mag that's the problem. Either you are very unlucky to have multiple Glocks that fail or more likely it's the ammo. I never manage to make my Colt Gold Cup totally reliable with Blazer JHP because the profile is like an ashtray!!!
Some times, strange things do happen. I have Colt Gold Cup 45ACP, Walther PPK and S&W 659 9mm. EVERY SINGLE ONE failed. I had to do a lot of work myself to make them reliable. The Gold Cup still is not completely reliable with the Brazer JHP because of the extreme profile. I had to polish the chambers, feedramp. Had to work on the angle of the extractor and also polish the ejector port to fix all of them.