The quickest way to give up on a diet is tell yourself you can't ever have something. I like the method where one meal a week you can eat whatever you want, cheat meal. So I think a brownie every once and a while is ok.