Yes, you have the opportunity to work on the sight on the way in. I always used to just relieve the bottom to put it farther down in the dovetail, but I finally bought the 60 and 65 degree files from Brownells, though not sure they are really required.
I watched some YouTube stuff and they really do not need to be all that darn tight, somebody, I think Ted Yost even advocates using pieces of (really) hard wood to do the drifting. I have just used brass or aluminum, and did grind a few of the brass drift punches to be flat on a side to fit the base of the sights. A layer of masking tape helps too.