He can't look her in the eyes, because she isn't really there. She's only an illusion, a magician's trick, a rabbit pulled from a hat in the second act, only to disappear for the grand finale. And he doesn't like magicians.
Slice a woman in two, pull a penny from somebody's ear, predict what card will be drawn from the deck. As a Doctor, he's had to sever memories from minds and rewrite the future with nothing more than a sonic and a slammed door. He wishes they'd just leave him alone. No more tricks. No more magic wands.
All the best tricks require an assistant anyway. A partner, someone to misdirect the audience's attention while the master works.
"Doctor, what is your name?"
Never look where the coin is. Look where it was, so they can't tell where it will be. "Stop this. Leave them alone."
Banter distracts, yes, but it also provides cues. River, I know you're listening.
Strax boasts, Clara screams, Vastra shoves Jenny behind her.
"Please, stop it." He doesn't beg. Not the Daleks, not the Cybermen, not the Weeping Angels— (Amy, he was begging Amy that time.) The tomb is sealed with his name, a word lost in the grinding lock of Gallifrey. There's only one time he could tell anyone.
Please.
(The coin appears, the card is revealed—-the assistant had it all along)
The door opens.
"Why did you open the door, sir? I had them on the run."
"I didn't do it." Always deny an explanation. "I didn't say my name."