Looks like shoulder/mount issues.
Took me a sec to realize you weren’t trying to tell me I was shouldering the rifle incorrectly, lol.
Food Court Apprentice
Semper Paratus certified AR15 armorer
Not sure if twist rate alone can account for a dramatic destabilization. I've shot 175's out of a 1:12 twist 24" barrel AIAT with easily sub MOA results (all between 100-200 yards though). I later got a 1:10 twist barrel for it and I did not see a dramatic increase (if any) in accuracy or consistency with both 168 or 175 SMK's.
This guy is shooting a 1:12 17" barrel AIAT at distance with 178's and seems to be doing fine: https://m.youtube.com/watch?t=182s&v=kKQ6qQAsSPU
"Gunfighting is a thinking man's game. So we might want to bring thinking back into it."-MDFA
Beware of my temper, and the dog that I've found...
If it was the twist rate, the 175gr groups would be similar with and without the can. As it stands, the gun shoots the 175's very well without the can. It's certainly caused by mounting the AAC.
You have this set up as direct thread, correct? Do you have any shooting buddies with .308 cans you can try?
IMO, the fact it shoots 168 FGM much better when suppressed suggests it may be just a matter of finding the right ammo (bullet design/weight/BC).