collective nouns - Can singular verbs be used with a plural possessive?

Is this correct?:

The group of students does their work well.

The group, the subject, I consider singular. When it comes to the possessive, though, I feel like it would be wrong to refer to a group of people as "it." Would it be correct to use "their?"

Isn't there some sort of rule against referring to a group of individual people as it?

And what about this sentence:

The family returns to their house. The collective noun, family, is singular, so would it be "its" or "their?"

@Peter Shor - Thank you, but in the situation where the sentence cannot be paraphrased in another way and it must be written formally, which would be used? (this was a question on an English grammar assessment). In other words, if the possessive was the only thing that could be altered, which is more correct?