To each his own. I shot IDPA w/ a laser (and thus not for score) for years because (a) that was my carry gun and (b) no amount of lonely range work can compare to shooting other people's COFs in public. Even though my time wasn't official it didn't keep me from wanting to beat other folks.
Draw is a little faster, reload is a little slower for most folks. At most matches you're likely to have more reloads than draws (10rd limit and mostly 12+ round stages) and my guess is that on balance the aiwb would have a slight disadvantage.
IDPA likely isn't against aiwb for competition reasons, though. The perceived safety aspect -- especially as there have been a few people who've shot themselves holstering their guns at matches -- is probably all the motivation they need.