It is increasingly common for the apostrophe to be dropped from the names of other institutions where the plural reference is a human reference - for example, Geologists Conference, Plumbers and Gasfitters Union. In all such cases, the plural word is not strictly possessive; its relationship with the following word or phrase is associative or descriptive, rather like an adjective.Now, a toilet block is not usually considered an institution. But, maybe I shouldn't always smile when I'm walking in Darebin Parklands and I see these two signs:


I always have an image in my mind of little 'mens' and 'womens' making their way into the toilet block. I don't know why they have to be little, but that's the way my imagination processes it.
If I visualise the toilets as a meeting place, a kind of informal conference centre, and think of the words as having the understood extra word, toilet - mens toilet, womens toilet - then it sort of fits the rule I quoted at the start of this post, ie it's not a place owned by men or women, but a place where they are gathered.
But, deep down, I want to add an apostrophe.