I don’t wish to be the bearer of bad news, Rosalin, but your intended dietary intake simply isn’t sufficient to support level of activity. As such, your body will hold on to the excess that you wish to eliminate, leaving you frustrated in the process.
While the removal of simple/refined carbohydrate from your diet is a sensible choice, you need to be incredibly astute with dietary intake, consuming sufficient calories to meet TDEE (total daily energy expenditure), while allowing the intensity of exercise and daily activity to create the deficit needed to burn excess energy stored in body fat.
As such, consult BMR/TDEE calculators to gain a clearer idea of how many calories your body requires, based upon measurements (age/weight/height) and level of activity.
Concerning diet, although you may consider yourself to be otherwise, even at 132lbs, you’re still incredibly slight, Rosalin. To ensure that calories are burned from fat and not muscle, you need to ensure that consumption of protein and complex carbohydrate each account for around 40% of daily intake and that healthy fat accounts for 20%.
A banana for breakfast simply isn’t sufficient to set you up for the morning. Instead, replace this with 2 soft boiled eggs and two slices of wholemeal toast or a protein shake (made with dairy-free milk) if time is short, while opting for a 40g serving of un-salted nuts as a mid-morning snack. A sandwich for lunch is a sensible option, but a bean/quinoa salad should also be considered to break up the monotony of sandwiches each day, in addition to providing greater fulfilment.
By all means, snack upon an apple or vegetables in the afternoon, since the natural sugars in the apple, for example, will provide you with energy upon hitting the gym, after which you should begin to re-fuel and replenish with another protein shake before settling down to your evening meal.