A palavra "mensalidade" em inglês é "monthly tuition" ou "monthly fee"?

Monthly fees for a course (or any other service - Netflix, etc).

Collins Dictionary states that Tuition fees are the money that you pay to be taught, especially in a college or university.
Back to your question proper: some prep courses use the expression "course fees" others mention "tuition fees". Universities, in general, are likely to use the words "tuition fees".
Tuition is a word that means training, teaching. In North America it took the meaning of payment for such training as well.

In addition to PAULO's answer:

tuition
>> aulas particulares.
>> mensalidade, anuidade [de faculdade, escola] (AmE)

Examples:

1) I was given an extra year of private tuition to get me into science.
>> Recebi um ano extra de aulas particulares para poder cursar Ciências.
>> Recebi um ano extra de aulas particulares para ser aprovado em Ciências.

2) The tuition is $3,500 per year.
>> A anuidade escolar é de $3.500.

One more (minor) grammar point: the word should be spelled 'month-ly' not 'montly tuition'.
See you around.