Given what I've read, if say it's extremely fortunate you're relaying the event to us as opposed to a news outlet reporting on either:
A. A homeowner killed with his own gun
Or
B. A nosy homeowner who took the law into his own hands (Zimmerman-style crucifixion).
All things considered, I'm glad you're unharmed and were not forced to plug that dirtbag. I would commend you on your restrain, however I feel it may be better described as deadly hesitation/reluctance. There were two points I'd say you'd have had a green light to put rounds down range, ESPECIALLY when he reached for the cell phone. That would've been the end of that little circus side-show (and the beginning of the three-ring circus). Whatever quarrels you have about possibly taking a life, you need to make peace with them if you're going to carry around a gun in any capacity. Had you shot him, it would've been fear-biting. Please get training and decide now what circumstances would force you into shooting someone.
Next, as others have recommended, take into consideration all of the repercussions of using deadly force in this situation (as well as future). In this incident, had you shot the male party, you'd be buying some local attorney a new sports car, whether defending yourself in criminal court or civil. I can guarantee the wife would've brought a lawsuit against you for shooting her beloved husband, whether he lived or died.
Depending on how your local prosecutor is feeling, they could make you out as a vigilante for traversing the considerable distance from your home to where the incident was occurring. As referenced earlier, take a look at the Zimmerman debacle.
If nothing else, view this as a solid learning opportunity. The more knowledgeable here could recommend the specifics to train up on.