I think that could work in principle, but I know that the meals I usually don't count would be something like dinner... and if you are someone like me, who likes to go out, the portions at restaurants are tremendous.... a single meal could have as many as 1000 cals itself if you include beverages and the like... on another day, it may be only 300 cals... so I think because of the wide range, it is best to try to pay attention...
I think a better idea is to give yourslef one day a week where you don't moniter your caloric intake... have one splurge day, but pay attention to the rest =)