Looks like Sheldon, Raj and Wolowitz were clearly too busy for their group dinner date with Penny. They have important things to do... like play Jenga!
Leonard: We need to widen our circle. Sheldon: I have a very wide circle. I have 212 friends on MySpace. Leonard: Yes, and you've never met one of them. Sheldon: That's the beauty of it
Leonard: At least I didn't have to invent 26 dimensions just to make the math come out. Sheldon: I didn't invent them, they are there! Leonard: In what universe? Sheldon: In all of them, that's the point