My XD .40 loves 180gr. It does well on 165gr, but does NOT like anything less than 165gr. As in I ran a box of 155/150gr (I don't recall which it was) and had two fail to feeds and a stove pipe. This is after ~2000 rounds.
I second AAShooter in that you should run a couple of boxes of different grains and see which holds a better pattern for you. And then make sure that your SD choice of bullets works the same at that bullet weight. I think it was a Winchester brand of hollow points at 165gr that shot completely different out of my XD than a box of 165gr Winchester ball.