When using apostrophes to show possession, does the object being possessed being singular or plural have any effect?

Not sure if I used affect/effect correctly, but anyway...

So, here's my question if the title didn't explain it well enough: say you have the following phrase: "in the men's bathroom". The plural of "man" is "men", so when showing possession with a plural word that doesn't end in "s", you use "apostrophe s", right? Well, what if "bathroom" were also plural? So you've got multiple men and multiple bathrooms. Would it be "men's bathrooms" or "mens' bathrooms"? Does the object being possessed being singular or plural make a difference as far as whether or not you use 's or s'? I could have sworn I was taught in school it does make a difference, but now I can't remember.