This. When I was in the range business, people would bring in pistols that weren't shooting center. We'd drift the sights and get them centered, then they'd get bent out of shape because the rear sight wasn't centered any more...
If you're not shooting actual groups, then it's best to leave the sights alone until you are. Your groups certainly could be smaller, but it wouldn't hurt to drift the rear sight a bit to the right to center things up.
Okie John