Black sights are awesome! I wouldn't run anything else-- and I used FO and hi-vis fronts for years. The sight picture is just so clean. And to put your fears at rest, my speed remained the same. A lot of great shooters run black sights. It's not going to hold you back, and may even aid your precision.
If you enjoy shooting steel, you owe it to yourself to switch. There is something magical about clean, black target sights on white steel.