modal verbs - Must in past. Obligation versus deduction

If you can use "have" as a modifier to make modals past tense, why can't we apply them across the board. For example May/Might in past "you might have known." Or "He must have been here." These are perfectly acceptable examples, but when we come to obligatory usage the structure is more important.

For example: "Do you have to insult my mother every time you see here?" Is interchangeable with "Must you insult my mother every time you see her?"

Conversely why can we say "Did you have to insult my mother every time you saw here?" This use in the past is grammatically correct and conveys an action that used to happen, but no longer happens, for instance the mother in question died.

The main question is why can't we say "Must you have insulted my mother every time you saw her?" I am a native speaker and I would understand that question without thinking the speaker was using my language incorrectly. This is from the perspective that as long as language follows syntactic patterns there is room for correct usages that may not be codified in books, but still can be used to convey meaning.