Whole foods are foods in their natural form. They're better than supplements because you're consuming the nutrients in the context they were meant to be consumed. I think it was shown that you don't get the same benefits from a beta carotene supplement as you do from eating a carrot. There's something in the whole carrot that helps you absorb and utilize the beta carotene in it. Whole milk is another example, although most people are afraid of the fat. The natural milk fat actually helps you absorb some of the nutrients in the milk.
Yeah, a calorie is a calorie, but there are good ways and bad ways to go about getting your calories. As I said, whole foods are high in nutrients that our bodies need. You will also see a difference in whole and processed foods in something called Glycemic Index. That's measures how quickly a given food item raises your blood sugar. If your goal is to maintain blood sugar levels or control fat gain, then you want to stay away from extremely high GI foods at times other than right after your workouts. The reason being, high insulin levels promote fat storage. If you're eating a ton of sugary, starchy things throughout the day, your body will be more likely to store fat with your extra calories. If you limit it to after workouts, then you're giving your body the anabolic boost it needs to build muscle immediately following a workout.
Another problem with "dirty" foods is that they lack the nutrients that you need. Often times, they will fortify the junk food with some vitamins to make it meet a nutritional standard. Enriched white flour is an example of this. As I said before, you absorb nutrients better when they're eaten in their natural form.
I don't see much wrong with chocolate as long as it's not packed with extra sugar. A small amount of dark chocolate won't hurt you, and they say that dark chocolate (70% or greater) is full of good antioxidants. But if you're powering down Hershey's bars and that kind of junk, then that's not good.