I'm glad to know I'm not the only one who went through all the same discussions and had the same aversions (no strollers!) as other folks..

I also went through all the same thoughts and concerns when my son was born. Switched from SFA guns to 1911s for the exact reasons you thought of. He'd frequently pick-pocket my flashlight and tried for my knife a few times. Keys were always gone from my pocket.

Things I did to mitigate issues with him:
-I did not perform administrative handling around him. I didn't want him to think that it was "daddy's toy".
-I did not leave any loaded guns off my body or outside of the locked safe(tumbler setup, no way to open by pushing buttons).
-My wife moved from off-body carry(her purse) to on-body carry...now she hardly carries(this is me rolling my eyes).

Things I learned:
-He still knew that I had *something* under my shirt that he wanted to see/touch/playwith. This always became an issue in public....
-Kids can still pull the trigger on a 1911. I remember reading somewhere on here that a LEO had found more than 1 child with a self inflicted gunshot wound because they point the guns into their chests, grasp with both hands and pull the trigger, which would mitigate nearly all the safety systems on a 1911.
-1911s get heavy when lugging kids around.

Where I'm at now:
-My son is now 5 1/2 and my daughter is 18 months...and at the same phase in her life when the wife and I had our "OMG we have to do something with the guns!!! phase". My son is curious about guns, but we're keeping same same practices of no lose guns around the house. He "helps" me clean and do maintenance on my pistols...but mostly on cleaning the WML that I keep on my HD Sig. We switched to DA/SA guns, but not due to child safety concerns. Him being involved with spring maintenance and cleaning the flashlight appeases his curiosity to the guns, allows him to be (what he feels) is hands on. We've gone over safety and what to do with guns and around guns. He never tattles, but came and told on a friend of his for handling a rifle (which was a BB gun), so I know those lessons have sunken in.
-I've switched to what @BobLoblaw mentioned; a low riding leather holster for EDC. I carry either a full size or compact pistol. I find this to be a great compromise between comfort/concealment and accessibility.
-I've been dabbling back with plastic SFA pistols in kydex holsters since we're doing more outdoors activities (best time of the year in FL!).

Hope that helps.