I'm sorry to hear that, that sounds stressful!
Firstly, I would speak to your OH first (if you haven't already), and ask him if he has noticed anything odd about his brother's behaviour towards you - he may not have noticed anything, so have examples to explain it. Maybe his brother is like that with everyone? Explain your concerns, but make it clear that you aren't trying to stop his brother from attending the wedding. However I think it is important to address this before you marry into the family!
If your OH doesn't have a clue, why not try and have a chat with his brother yourself. Explain, calmly, that you aren't sure what you have done, and ask him if he would mind explaining why he seems to have taken a dislike to you.
Regarding the friends, that's not cool. I'd be fuming if I was compared to my fiance's ex! You have to remember that they aren't together for a reason, and he has chosen to marry you. Have a chat with your OH, and hopefully he should reassure you - maybe they made a thoughtless joke that didn't come off?
If they continue to be rude, I think you're within your rights to at least ask your OH to have a word and tell them to be on their best behaviour from now on. If they still make snide comments, I would ask for them to be removed from the guest list, if they aren't even going to respect your OH's request.
Obviously we are only getting one side of the story - I'm assuming you've made attempts to get to know all these people? Have you and your OH been together a long time? If not, they may simply want reassurance that you aren't going to hurt their friend. If you have, then maybe they need to get their head out of the past.
Try to stay calm and reasonable throughout all of your conversations with your OH and the others - and remember, that's only 3 people at your wedding. Surround yourself with other friends and family that love you, and you'll barely even notice them! Worst case scenario, speak to a couple of friends that you trust to remain cool and ask them to keep an eye out for any trouble from them, so that you don't have to think about it.
In my experience though, people are usually on their best behaviour for weddings!