Silicone spray does not seem to attract dirt like most “gun lubes” do. Sure, I’ll find some lint on a pocket carried gun now and then, but it is never really a problem. Since I stopped relying on cotton shirts to cover my gun and switched to a nylon vest or silk Hawaiian shirts - no issues. IWB guns.
ETA: Silly putty/“flubber” similar slightly stick gel/pastes are nice for removing lint and dirt. Better than compressed air and a brush, actually.