Actually it makes a huge difference... depending on what tires you've got in the first place.
Biggest problem isn't the tires though, it is that between the additional speed and the decrease in tire diameter after you go to slicks, sometimes you will max out your gearing on your mountain bike. I mean, think about it, you're running a 44t gear at most up front on a MTB, with 26" wheels. Basically most road bikes are going to have maybe a 52t on 29's. Big difference there. Also you're taller than on a road bike, so when you're in a pace line, you can feel your head and shoulders getting out of the slipstream.
If you are by yourself and not in a pace line and not going nearly 30 mph, it's not a problem, you really just need to find some slicks. Also keep in mind that some companies such as Hutchinson make tires such as the Top Slick, which are basically road bike tires that go on a mountain bike, and these are way fast, but the side effects are mentioned above.
As for the tire pressure thing mentioned above... well... first of all it's just not a big deal with slicks, but second of all, if you do get something like the Top Slicks, hell, you can air the things up to like 90 psi.
Last but not least... keep in mind that most mountain bikes will in fact accept road bike wheels. They even make road bike wheels with discs, assuming you have discs, cyclocross bikes do this alot. What I'm getting at is that not only will your gearing take a hit, but it's a PITA to swap tires. A good middle ground would be to get something like a road bike wheelset, put on a Shimano 105 cassette, road bike tires, and stick it on your mountain bike. That way you get all of the benefits without as many bad side effects.
Personally, I'm selling my road bike tomorrow. I can keep up with all but the fastest of the local roadie snobs on my nearly 6" travel 35 pound heckler on 2.1's. I can pull a pace line at 22 mph just fine on it. Owning an additional bike simply isn't worth it unless you're the kind of guy that just HAS to go 30 mph in a pace line like a roadie snob. It's funny to see roadies disagree on this. Yes, road bikes are a little more EFFICIENT, and that's it. Mountain bikes will always be more comfortable, and quite honestly, you can go just as fast as most roadies... you'll just be putting out a few more watts than them, yet they think they're the real men for some reason. They'll probably even have matching clothes while doing so.