I now have a bunch of bent brass punches, because the sights were still so tight after filing and I was paranoid about filing them too much more and leaving them too loose. So instead, steel punch.
Ugly? Yes. Shoots well? Yes.
(yeah, I know I'm opening myself up for all sorts of jokes with that line...)