grammaticality - "How big of a problem" vs. "how big a problem"

Quite a few phrases in English are constructed like so:

How [adjective] a [noun]...?

This is the question form of the construction, which is often answered with the negative:

Not that [adjective] a [noun].

or the positive:

Quite [adjective] a [noun].

However, from time to time I'll hear the word 'of' inserted before the 'a', e.g.:

Not that [adjective] of a [noun].

This usually sounds wrong to me, with the exception of the case where the adjective 'much' is used. So, this sounds fine to my ear:

Not that much of a problem.

whereas this doesn't:

Not that loud of a noise.

Why is it that 'much' should be used with 'of', and other adjectives not? Is it because 'much' is seen as measuring a quantity (of something), whereas other adjectives that may be used in this construction are seen as measuring the quality of a whole thing?

Answer

What about "not that high (of) a fence"? "not that red (of) a heart" "not that smart (of) a person? not that big (of) a problem?

I would argue that if you use the word that to qualify the adjective, the of conveys the meaning of comparison of a specific entity to the class of general entities to which it belongs.

I'm sure the usage can be regional, as well. There is no hard and fast rule.

EDIT

Hey, I did some more research. Dictionary.com has the following usage note for "of" :

Of is sometimes added to phrases beginning with the adverb how or too followed by a descriptive adjective: How long of a drive will it be? It's too hot of a day for tennis. This construction is probably modeled on that in which how or too is followed by much, an unquestionably standard use in all varieties of speech and writing: How much of a problem will that cause the government? There was too much of an uproar for the speaker to be heard. The use of of with descriptive adjectives after how or too is largely restricted to informal speech. It occurs occasionally in informal writing and written representations of speech.

So, I suppose that's the reason why adjectives other than "much" combined with "of" sound odd to your ear. I believe "that" can be included with "how" or "too" in this synopsis. Replacing "much" with another adjective occurs occasionally in informal writing and in speech, but isn't unquestionably standard.

When I say these constructions out loud, to me, I often want to insert the of but perhaps that has something more to say about the informality of my speech rather than the correctness of the construction. :)