grammaticality - Is this statement, where the same pronoun refers to two entities, grammatically correct?

I had this conversation with a friend earlier:

Friend (to a group): If you don't like [this show], there's something wrong with you.
Me (to friend): Or you're just a little more mature than you and me.

In my statement, you switches from the other members of the group (or maybe people in a general sense) to the person I am talking to. Is this okay from a syntactical perspective?

I can't think of any other examples where the same pronoun switches people without using a noun in between. Is there something in this statement that makes it obvious the pronouns have switched what they're referencing from a pure grammar perspective?

Edit:

To clarify, what I was trying to say in my response is "We both like this show, but that's probably because we're both immature. The other people in this group might not like this show because they're more mature than us, not because there's something wrong with them."

Answer

To clarify, what I was trying to say in my response is "We both like this show, but that's probably because we're both immature."

TL;DR: What you are trying to say is (referring to your friend):

Or you and I are just a little less mature than the others.

Let's analyze your sentence in different cases, and explain why it's wrong.

Or you're just a little more mature than you and me.

• The first you is the second person plural.
  In that case, the phrase "you are just a little more mature" would mean that all the people present except yourself are a little more mature, including your friend.
  Then you say "than you and me". If that you is again the second person plural, then you are saying that all the other people are a little more mature than all the people, excluding yourself, and including your friend; if it is the first second person plural, then you are saying that all present people except yourself , but including your friend (the one to which you are speaking), are a little more mature than you and your friend. In both the cases, there is somebody that is a little more mature than himself/herself.





• The first you is the second person singular.
  In that case, the phrase "you are just a little more mature" is referring to your friend (the one to whom you are speaking).
  Then you say "than you and me". If that you is the second person singular then you are saying that your friend is a little more mature than you and himself; if that you is the second person plural, then you are saying that your friend is a little more mature than all people, excluding yourself and him. Again there is somebody who is a little more mature than himself/herself.

Your sentence is not ambiguous; it is simply wrong.