grammaticality - "My Mom" vs "Mom" Usage

Context
My brother and I when having a conversation that refers to our mother usually use "my mom" to refer to her. For example "Have you talked to my mom today?" is a common question we ask.

Now the discussion came up when some of our friends pointed out that it was weird and incorrect for us to say "my mom" since we share the same mom and are siblings and that we should just say "mom".
However, my brother and I think its grammatically correct to use "my mom" since its technically a factual statement.

I'm just trying to find out if our usage is incorrect in English grammar and if our friend is correct that we shouldn't use the pronoun "my" in conversations between siblings.

I just wanted to clarify that while English isn't our native language, my brother and I have notice that we also don't use the correct possessive pronoun in Spanish. Now what I'm getting from both of the answers posted is that this is more of its technically correct, but it might confuse other speakers as its not common use.

Answer

Mom and my mom are very different: it is not simply a question of possessives.

If you and your siblings shared, say, a truck, then you would simply use the appropriate pronoun:

[to stranger]: Where did I park my truck?

[to sibling]: Where did I park our truck?

Of course, you could apply the construction to mom:

[to stranger]: Have you talked to my mom today?

[to sibling]: Have you talked to our mom today?

This is still a bit unusual, primarily because Mom is often used as a proper name. For example, you might say:

Hi, Mom. It is so nice to talk to you!

However, the name Mom is usually used within one's immediate family. I might say "Have you talked to Mom today?" to my brother or even father. This is despite my father, clearly, having a different mother than myself. But I would not use the name Mom when talking to someone outside my immediate family; instead, I would say my mother.

Of course, this last point is not a matter of grammar. The question of who is allowed to use a term of endearment has no definitive answer; it depends on personal preference and cultural norms. Even within the English-speaking world, different people may have different standards for who uses the nickname Mom. To the best of my knowledge, my analysis is fairly standard for white Americans in the Mid-Atlantic. But other groups may have different usages.