I think it is valid to consider factors other than how well you shoot a given platform. That is certainly high on the list, but it is not the entire list. I shoot my 9mm PPQ more accurately than my Glock 19, but I shoot and carry the G19. It's a better size for carry, it is safer AIWB with the use of the gadget, I have many more magazine and holster choices, I feel more comfortable working on them, etc.
You will get better with any platform if you practice with it. Going back and forth between the two might be holding you back on the M&P.