Hi T@ta e Josneywat,
"You will see 'dinner' used with both- it depends whether you think of it as an occasion, like a dinner party, os simply the act of eating."
They've invited me to tea = a meal at about 5pm
They've invited me for tea = a cup of tea"
Acho que "tea" é a mesma situação como "dinner" (pode usar "for" or "to")
Se a gente queria especificar "a cup of tea" falariamos
They've invited me for a cup of tea