This is a quick late night post but on a subject that probably would deserve more time. What makes a face look male or female? How to make a face appear to be from the opposite gender? This article (in french) gives a few answers. It starts with "Defining what sex people pertain to is one of the tasks that our brains does the most efficiently". And still, we don't really know well how our brains do that exactly.
The study is based on an experiment with 30 male and female subjects exposed to over 300 photos of Caucasian faces. It gives a few ideas of what in a face can indicate gender. Indeed, it's not the haircut, beard or shape of the eyes that give the best information, but instead the luminance of the eye and mouth area and apparently the overall green/red colour constrast of the face. The shape of the jaw and skull also are quite important indicators.
Based on that informartion, I did a quick test on a photo of my grumpy face. I tried to make it look male while altering it as little as possible. Quite hard, but quite interesting.The opposite would probably be even more difficult in my opinion.