I think a substantial part of what I emphasized in bold is a matter of making sure the stock fits you extremely well for field shooting, minimizing recoil through mechanical means (to allow you to concentrate on the fundamentals) and the right optic. I realize that probably sounds obvious, but sometimes the forest gets overlooked for the trees.
A McMillan Sako Hunter pattern stock works for me (and apparently many others due to its popularity). It has both a high cheek piece and a slight palm swell, which aid in getting a natural, repeatable shooting position.
The T3 .300WM pictured above was set up for longer range shots on big game in less than ideal conditions. It is not as fun to pack as my favorite little .308 carbine, but it will get the job done at extended ranges, and it beats the heck out of carrying a 15.5# chassis rifle.
Previously I had set my steel IPSC steel plate at 450 to confirm dope, and the data was on, so I found a spot on a rock face and sent 3 quick shots.
Bearing in mind that I only shot one group, and that really (statistically speaking) doesn't mean too much, but the 3 rounds dropped into about 2&3/4"s at just past 600 yards. That is under .5 MOA for a hunting bullet. However, all the shots on paper averaged out to a more realistic 3/4 MOA.
For me, I will take 3/4 MOA from a magnum chambered, sub 10# hunting rifle any day.
For what you seek, with a max range of 500 yards, under ideal conditions, I don't think you could go wrong with a T3 mag, dropped into a McMillan, with a muzzle brake. No, it is not a high end custom rifle, with all the bells and whistles, but it will fill the freezer.