You dont lose fat by not eating fat. Nuts contain good fats, poly and monounsaturated. It is recommended that between 20-30% of your total caloric intake should be fat.
Just remember that one gram of fat has 9 calories, where carb and protein contain 4. Walnuts, almonds, and pecans are a great addition to any diet.