A couple things I forgot to mention:
My brother and I are not close at all. In fact, I barely consider us acquaintances. We had a falling-out when I was a teenager, and I avoided him for years. Now we’re on “ok” terms and see each other a few times a year (holidays), but that’s about it. So I personally am not bothered by the fact that this guy is my brother’s friend, and my brother wouldn’t be “protective”.
Not only are we not emotionally close; we’re not physically close. My brother lives a few hours away from me and his friend. So I’m not cutting into their “hanging out” time by hanging out with the friend. I think they see each other once every couple of months.