Indeed, collocations-wise, you would find the entry "to the contrary" in good dictionaries and grammar books. Not "to the contrary of".
As an example you can take the following sentence: "The brown horse didn't beat the black horse. To the contrary, the black one won."
"Unless you have specific information to the contrary, it is obvious that she is lying.'
In some sentences "of" sits well with "to the contrary" particle, as in:
In the second half of the 70s, the debate resurfaced with the need to explain the
nature of the transformation because, to the contrary of some hypothesis, capital had
dominated economic activity in the rural area. Modernization was induced by the State
and resulted in the deepening of the inequity in land distribution, revenue, and power.
Population exclusion reached high levels, generating rural-urban and rural-rural population
flows. Agrarian conflict and violence multiplied. Environmental impact was remarkable "
In other words, one has to be careful with its usage, with time you will know when you can use the preposition or not. You will get the hang of it.
As for "to the contrary" doesn´t necessarily means or it´s interchangeable with "as opposed to"
Just for illustration´s sake, let´s take the two following cases:
To the contrary of false accusations, Mr. Oktar has no criminal records. Home / To the contrary of false accusations, Mr. Oktar has no criminal records.
The scientist can make proposals but he/she must be open to and even welcome evidence to the contrary of current theory. One instance is, the old theory that the earth was flat, then scientists studied and found evidence to the contrary.
(not exactly the opposite, a round object doesn´t exactly means the opposite of a flat one. Albeit the notion of two opposite schools of thinking here.)
They are impatient with other children who do not know what they themselves know (“Don’t you know ANTHING?”) and can become enraged with a child who maintains something to the contrary of their own views. Failure to know what you are supposed to know, or worse, to believe something which contradicts the peer-group can lead to conflict or ostracism from the group with considerable unhappiness as a result.
(here not opposed in nature (black vs white/if you are not with me you are against me etc); but "opposition", something that contradicts the previous belief/knowledge/thesis/what you think/the public thinking etc)