Since I'm not 100% sure which machine you're referring I can't make any absolute claims on the exercise in question. But I do know that with squats, deadlifts and lunges (supposedly the big 3 of glute builders), it's very easy to leave the glutes out of the exercise. The glutes get the most work in hip hyperextension (not to be confused with lumbar spine hyperextension), thus exercises like glute bridges and hip thrusts, which involve hip hyperextension in a position where the hamstrings are weaker than the glutes, tend to do pretty well at building the glutes. It sounds like the machine you're describing sets you up to do a movement like a 1-legged hip thrust, only face down. If you know that you're working your glutes hard with the exercise (and if you are, you will know about it), then the two most likely problems I can think of are:
1) Low frequency training. I'm not sure how often you're doing the exercise. If you're doing the whole body-part split thing that most muscle mags recommend, then that's going to give some rather slow progress, unfortunately. Try doing glute-specific exercise 2-3 times per week.
2) Not eating enough. If you aren't in a calorie surplus, you won't usually build much, if any, muscle. Could require some bulking and then cutting to get the butt you're after.
Other than that, does this machine have you doing the lifts with straight legs or bent knees? The straighter the legs, the more you'll use your hamstrings. If your legs are bent, however, it'll be easier to focus on your glutes. And you may benefit from different rep ranges.