Hmm, did you know that President John F. Kennedy is considered to be one of the most fashionable President?
Fashion in this case has to do with being best dressed, stylish, etc. Even if, nowadays, rappers are fashionable and sometimes not meaning best dressed (as per the old definition of fashion and taste).
It´s that fashion is way much associated with women, as the popular wisdom that goes "men buy and women shop". So, they (men) even may buy a stylish or sophisticated tuxedo or something but it´s more associated with money or power (and/or practicality) than pleasure per se.
Other thing with women´s fashion is that it´s associated with changing, whereas we don´t see much dramatic changing on men´s clothes. There are, but at at a much slower pace.