How does it wear on you...
I think there's either in some cases people don't shoot enough to be truly comfortable around the tool or just have a mindset to be carrying.
Don't get me wrong I don't advocate throwing a gun loose in the pocket, but secure in a iwb holster, the only real conceivable problem is being careless upon unholstering. And if you are clumsy, then having a safety isn't going to prevent someone from making the oopsie they are afraid of.