Olivia takes a case which takes her away from Fitz and the White House just as the story of their affair breaks in the media. Fitz will no doubt take this as a sign of rejection while Olivia will see it as an act of self-preservation.
[to Fitz] You placed Olivia in a dangerous position. You outed her. You thought by throwing me out of the White House and moving her in you'd be making this grand gesture. Finally she wouldn't be just a mistress anymore. But the minute she stepped foot through these doors, the minute she moved in here, she became what we all become when we live here and what is that? Yes, a statue. Inhuman. Discussed from every angle, viewed, exposed. She's not just a mistress now. She's America's mistress.