What is wrong with me?

You sound perfectly normal. Just arrange for very short visits with people and always have an escape plan. When I used to go to parties I never went with anyone else, I always had my own car to make a getaway at the appropriate time.