Pistol mags and the bullets in them, in my view, are subject to a lot of abuse, though I should clarify that I’m mainly talking about mags in carry guns or that are carried as spares. Those ride around pressed against my body all day and move around with me, which also means going through all the climate and humidity changes I do.
I agree that with a mag that sits and does nothing, the ammo isn’t suffering any more abuse than boxed ammo, so I guess my question RE changing out rifle bullets is only relevant if it’s due to handling the ammo if and when you rotate out the mags.