of course it matters. say you're on a cutting diet...so 500 calories below maintenance every day. So all week long you total a cutback of 2,500 calories.
Then on saturday, in addition to your normal 'clean diet' you decide to binge and eat a half a medium pizza, and 6 cans of beer. That's like 1900 calories if its Pizza Hut, thick crust, and non-Light beer.
if you then drink 2 cans of soda on sunday, and have pork sausage with breakfast, you've obliterated the calorie deficit you created all week, in just two meals...that some wouldn't even consider 'binging'.