You are pretty much describing me. What amazes me is the number of people nowdays who don't - or won't - get that "just want to be left alone" part. Some days I feel like I need to deliver a slightly more conciliatory and updated version of "J.B. Books' dicta" (from Glendon Swarthout's "The Shootist") right to some doof's face, but haven't done so. Yet.
I suppose I will automatically try to do what absolutely HAS to be done if and when it comes to that, but "defend any and every victim" stopped being a thing with me long ago. I've seen too many situations - none involving the discharge of a firearm, thankfully - where noble intentions got the stuffing taken out of the noble intender.