You have to know when it is time for you to make your move and you have to fully commit yourself to that moment without hesitation. When you have come to peace with this, you will come to peace with your fear of ever having to defend yourself.
Self-defense is a simple part of all animals lives except human beings. We doubt it, ignore it, evade it and more than anything, fear ever having to use it. Just so you know, THIS IS NOT NATURAL!
Every creature has a defense mechanism including human beings. We can club with our fists, elbows, knees, and feet. We also can scratch, spit and bite. But most importantly, and unlike all other animals, we can incorporate other objects outside of our selves as weapons.
Have confidence in yourself as a true animal of earth with the abilities to fight back naturally when needed and you won't be any attacker's victim. They will be yours. If it ever comes to it, JUST FIGHT BACK!