grammaticality - Usage of apostrophe in “baker’s dozen”

In the phrase “baker’s dozen”, why does the apostrophe indicate possession of a (single) baker? Shouldn't it indicate possession of all bakers in general? Shouldn’t it be “bakers’ dozen”?