Sex is more than just a physical response. Your emotions have something to do with it, too. When your mind is too stressed out to focus on sex, your body can't get excited, either.
Lots of different worries can lead to the problem:
These things may lead your body to release stress hormones like epinephrine and norepinephrine.
Fear that you won't perform well in bed and satisfy your partner sexually
Poor body image, including concern over your weight
Problems in your relationship
Worry that your penis won't "measure up"
Concern about ejaculating too early or taking too long to reach orgasm
Anxiety about not being able to have an orgasm or enjoy the sexual experience