I'm also 24 (although not for long). I think I got over the party phase of my life right around when we all grew pubic hair and suddenly fairy bread wasn't cool anymore. In other words, I never got into what the average 13-23yo calls "fun." I did drink a bit as a teenager (at home until I was 18), but stopped drinking altogether about 4 years ago, because even though I didn't have any compelling need to drink regularly, once I had alcohol in my hand it was unlikely that I'd stop drinking at any reasonable time -- the first inhibition to go when under the influence was always the inhibition against another drink. I do go out on occasion, but only with people who I like when we're sober, and as a sober person. People who just want to be drunk and get me drunk will either re-evaluate their own attitudes towards drinking around me when they realise that I'm not participating in the drunkenness, or they'll lose interest in having me around, if I don't lose interest in them first (in saying that, if you live to drink, then it's unlikely we had any common ground to begin with, so such a person and myself won't feel like we're missing out on much by not hanging out together).
As an adult I've become annoyingly Christian, which makes it easy for me to find people who aren't interested in drunkenness, although not so easy to find people who just don't drink, period. That's okay. People who can handle their liquor and have a few drinks without turning into arsehats are pleasant company from my POV. People who are arsehats despite being completely sober are also good entertainment. Outside of my Christian circles, people who don't live to drink are harder to come by, but they are around.
I'm a uni student at the moment. Here in Australia, the university culture (at least insofar as I've experienced) appears to place the occasional activation of the grey stuff between our ears ahead of demolishing said grey matter with alcohol, but that might just be that I gravitate towards people who exhibit subtle signs of being the same kind of weirdo as me, so it might just be that I'm super-effective at phasing out an awareness of people who I wouldn't be excited about knowing in the first place. Nonetheless, I seem to do alright at meeting people at uni whom I could stand to spend time with outside of uni.
I'm also a D&D nerd (the exact sort of person you'd expect to compete in powerlifting and be into fitness). The friends I make over D&D tend to overlap between my school and church circles, so maybe that doesn't help, but I've never played D&D with someone who, at the time, was heavily into drinking, even in high school.
My best advice would be to think about what hobbies and interests you have, rather than to focus on the party-obsession that you don't have, and get actively involved in groups who participate in those same hobbies and interests. In general, when people are focused on things other than drinking, they seem to do so either sober or with a sensible amount of alcohol in their system. I don't know what your interests are, and maybe you haven't thought about that, either. We both know what one of your interests is, and that's fitness. Ever notice how at the gym, most people are sober? This is a relevant starting point. Talk to people. The people whom you feel like talking to ever again are more likely to be like-minded. You probably have some other interests, though. Maybe they're activity-related, such as a sport, rock climbing, hiking, fishing (okay, that one's not all that active, unless perhaps you're using a spear), camping etc. Maybe it's intellectual, such as reading, writing, biology, puzzles, trivia. I don't know. But I do know that in my experience, the more you focus on living out a specific interest (or groups of interests), the less time you have to be blind drunk. And the more you do that socially, the more you'll find yourself hanging around with people who aren't obsessed with drinking, either.