Verb + to infinitive or Verb + ...ing

Is there a general rule whether to use the Verb + to infinitive or the Verb + ...ing format?

There are cases in which I can't decide which one to use.
Like:
-They can't afford to go out very often.
-They can't afford going out very often.
Or:
-I don't mind to wait.
-I don't mind waiting.
Or:
-It was a nice day, so we decided to go for a walk.

-It was a nice day, so we decided going for a walk.