Music and grammar doesn´t necessarily go hand in hand, and - for effect - the lyrics tell a story from back to the beginning.
Anyway, from the beggining, one can stablish that there are two characters in the story and it´s no "me" that is pulling up in the fast car, it´s him.Swinging in the backyard
Pull up in your fast car whistling my name
(my name refers to "me").
So, it´s easy now to establish who pulls up in the car.
But it´s music, so it´s implicit and works well this way.