After posting I did several more beer pitchers full of 5.56 cases, and I think it is maybe even better than I initially thought!
My knee has been being a pest, so I am sitting on a stool that positions me above the handle at the bottom. My right shoulder has been being a pest (I am a mess!!!!) so I was trading off between arms, and putting the second hand on top of my hand on the handle for the last ooomph that is actually swaging. I got in a groove and it was working great and the handle was much more stable.
ETA: I thought about gluing them together with JB Weld, but I am glad I didn't bother and make that mess. I think tapping the hole might have helped, and I am glad it worked as well as it did, cause I put a pretty big dose of red Loctite on both the treaded hole and nut...
ETA Again: While I had the tools handy I did my second APP.