Yes, this is a photo of Peter on a horse. White Collar will flash back to this character's past in January 2011.
This is a shot from "Burke's Seven." It's the first While Collar episode of 2011.

White Collar Season 2 Episode 10 Quotes

Neal: Do you know how much the FBI could benefit from me being able to fight?
Peter: I think New York is safer if Neal Caffrey is lacking in at least one skill set.

Mozzie: It's worse, I'm in the system.
Neal: Not quite.
Mozzie: Ivan Bliminse? The invisible man? That's appropriate.