Your loyalty to the people whom you love is always more important than whatever lingering loyalty you may sometimes feel to other men's judgment on your manhood.
Most men feel that they receive love and therefore know what it feels like to be loved; women often feel we are in a constant state of yearning, wanting love but not receiving it.
Whenever a single woman over forty brings up the topic of love, again and again the assumption, rooted in sexist thinking, is that she is "desperate" for a man. No one thinks she is simply passionately intellectually interested in the subject matter.