negation - Answering negative questions with "Yeah."

I know I need to answer in the same way regardless of questions are positive or negative in English.

For example, when I didn't go to work yesterday,

Q. Didn't you go to work yesterday? (Did you go to work yesterday?)
A. No I didn't.

I am still taking time when answering negative questions in the real life, because in the language I was using before, it was deferent way. If I say yes to negative questions, it means I agree asker's negative questions. Anyway, I have been adapting to the English way to answer.

But what confuses me is that I meet those kind of native speakers who answer "Yeah." to the negative questions.

For example,

Q. You don't want to go?
A. Yeah. (When he/she doesn't want to go.)

There have been many of native English speaker answering like this. So I am wondering if there is a difference between 'yes' and 'yeah'.