grammar - please look at this sentence about usage of would in the past

Can you take a look at this sentence:

"it was 4th of June, and the day after that was my grandfather's birthday, and he wouldn't guess that we were going to buy him a present"

My teacher says its correct, but I say we should say:

"he wouldn't have guessed"

instead of wouldn't guess, it's not a habitual or regular even so we cant use would as the past form of will, I'll highly appreciate if you clarify me about this sentence and usage of the "would" in these kinds of tenses.