It's tough, because it is your day, but if your sister is in a long term relationship with this guy, I guess he probably should get an invite.
I would probably invite him, but state to your sister and your parents (and maybe him!) that if he causes trouble, he will be asked to leave. Then brief your groomsmen and venue so that they can keep an eye out and evict him if needed.
That way you've given fair warning. He may even decide not to come as he knows he won't really be welcome. If he does, then great, you'll have been the bigger person.
Another option - and this really depends on the size of the wedding - is you could have close friends and family to the day and invite him to the evening?
Finally, can you maybe try and spend some time with him - maybe you and your fiance go out bowling or something with him and your sister. I don't know how much time you've spent with him, but maybe just make one more effort. Even if you don't get on, you might find that you can at least tolerate each other, which might help reduce your stress about it?
I hope this helps, good luck! X