Your question should be dubbed "let go of someone/something" vs "Let go of":
The former, meaning "to stop holding someone or something" Here physically or Etymology: based on the literal meaning of let someone or something go (to stop preventing someone or something from leaving).http://idioms.thefreedictionary.com/let+gohttp://www.macmillandictionary.com/dict ... someone-go
Let go has also that (physically ring to it) sense: Also, let go of. Release one's hold on, as in Please let go of my sleeve, or Once he starts on this subject, he never lets go.
So, to my thinking both can be used physically or in abstract ways (non literal ways).
Let go of (to me), has physical sense or that of really holding someone forcefully (even in relationships), or that the person don´t want to (that is, forcefully wants to take hold of his/her espouse/idea/life standard etc, etc).
"let my boyfriend go" certainly is more colloquial, hence the higher use. (my educated guess here, though.)
The second fact is that these days, people don´t are so attached to each others when it comes to relationships. Well, part of the explanation, today even families are not that connected as they were two centuries before. Or do they?