As your brother I want to say
you have nothing to worry about...

But you both were
expected to marry each other from his birth on.

But do you really know him that well?
As far as I know both of you never met alone.
Do you know his opinion on you.
Does he know yours?

Although female kin is quite rare.
It has quite possible he socialised with some of them.
And rather wants to know them better then you.

The thing is, as long as you two marry there will be no problem.
Learn to grow to adapt to each other and compromise.

No! He is mine!
No sharing! I won't stand for it!

Then your changes to win him
over has plumbled.
Even if the rumors are untrue.