Using "have difficulty" in that sentence...
I have difficulty understanding TV programs in English. (correct)
Yet, if you want a preposition, you better to use "with" as in (that is, when you have difficulty, you have difficulty with):
Children may have difficulty with understanding the meaning of words and concepts.
This difficulty with reading is called dyslexia.
I have difficulty with understanding TV programs in English.
Back to the initial question: if you had used the word "difficult" on the sentence, you could use the prepostition "to":
The French speaker is very difficult to understand, and people would tell me that...
Anyway, if you could take the liberty to change or leave out the word "difficulty" you could use the sentence Marlon offered, it´s also grammatically perfect.