Well, that is because, by your words, you're using SOME of the sight... but most of your hand is gripping the slide, looks to me. And THAT is what _I_ do it with.
Possibly, some of the newer manufacturers have been truly better for the experience/ tough keeping of those optics. Cannot tell what yours is, but I've had four different brands on the three G19s, not for a couple of years now, so I probably should have kept quiet. I can tell for sure that your sight in that photo is a lot bigger than the now-out-of-production sight that my one remaining Glock custom pistol has... which means YOUR sight would be somewhat of an issue to the holster I had made for my three pieces back in 2009, and still carry Glocks in that modified holster with regular fixed open sights.
Using a lot of finger duty on my three optics- the other "good one" was an RM02- was practically ALWAYS getting crap on the visible lens when cycling the slide... usually on the front side, but sometimes on the back too.
.