Hi there, Leonardo.
We must admit that listening is sometimes a tough nut to crack (not easy sometimes), and some words aren´t really used everyday, all the time. On top of it, the way they pronounce, the ear not being used to some patterns (say, the fact of English word order being somehow "reversed" if compared with Portuguese).
Add to it the sound of words changing in special cases ("at" -a sound, "all'' -other sound "at all"=a whole new pronounce, as if it was another word, sometimes the very particular sounding of vowels (a, i, o, that can be voiced in unusual ways to Brazilian ears, along with the variety in accents (American, Australian, British, Africans, Indian, etc etc)
It would took a whole year to talk about some drawbacks, but I am going to let you on a little secret. When you watch a movie be aware that they translate the language, culture, and then the subtitle may be very loaded with plenty of meanings, be translated by way of "senses".
Sometimes watching some series, let´s say The Simpsons (on cable TV, or in Portuguese with subtitles), will see the character saying one thing and what is subtitled is a different version.
Nothing to sweat about, though. When that happens, continue on watching, and use it as an exercise to see how many idioms, how many words, expressions you have learned...
With time you will notice that your English have improved a lot. Practice and repetition (*) makes perfect.
*I mean, not repetition of a movie, reading etc; but the fact that you will meet some expressions all the time, if you somehow is in touch with the language on a regular basis.
The repetition I refer to is that of some sentences/expressions given some particular circumstances (hate, happiness, surprise, worry, and so on.)