usage - Would you say 'yes, neither do I' / 'yes, me neither'?

My question does not have to do with the correctness/incorrectness of 'neither do I'/'me neither', but with the presence of the 'yes' (or 'yeah', which is how it most often 'comes out' for me) at the beginning.

If someone were to say:

I love chocolate.

I'd naturally answer (orally) one of the two:

Yeah, so do I. / Yeah, me too.

But if someone were to say:

I don't like driving.

Would it be normal to answer:

Yeah, neither do I. / Yeah, me neither.

Or with it be best to say:

No, neither do I. / No, me neither.

---

I do not understand why my question was marked as a duplicate. My question is not about the personal pronoun (I or me) but about the adverbs (Yes/Yeah or No). I do state in the first paragraph that my question concerns "the presence of the 'yes' (...) at the beginning.", whereas the question I'm supposedly 'duplicating' is clearly focused on the personal pronoun.

I shall edit to make the words I'm concerned with in bold to see if that clarifies that I'm not interested in the pronouns.

Answer

The short answer is yes, it would be normal to respond with either statement.

As was mentioned already in the comments, the "yes/yeah" parts of your example sentences are simply agreeing with the original speaker. It would become awkward if they'd followed their statement with a question ("I don't like driving, do you?" "Yeah, me neither.") and sounds more like you'd formulated your response before they'd finished speaking, but as it stands it's not awkward and definitely not incorrect.

I know you didn't ask about the difference between responding with "me" or "I" but in formal speech they should both be "I" since you are the subject of your own statement. "Me too" and "me neither" are both fully accepted in conversational speech, though.

Having grown up in England and then moved to the States I will say that I have (on incredibly rare occasions) encountered people who are amused by my "formality" when I say "neither do I" but have never encountered the opposite with "me neither."

So, while both are fully acceptable statements in both the English speaking countries I have any experience with, the States appears to have a (very, very) slight leaning towards "me neither" in an informal setting.