it has fair amount of saturated fat.
If saturated fat is a concern for whatever medical reason (i.e. you have or are genetically prone to conditions that are aggravated by high saturated fat intake), you could use skim milk (or dairy products made with skim milk, like skim milk yogurt) instead of whole milk.
Lactose in milk may be more immediately bothersome, although some people affected by lactose malabsorption may be able to mitigate that by using lactase enzymes and/or limiting the amount of milk drunk at one time and/or using fermented dairy products like yogurt and cheese.
Of course, if you have a true food allergy to milk, you obviously should not drink it.
Soy milk per se is not bad in ordinary amounts, though if you like unsweetened varieties, that would be better than the sweetened varieties (of course, if you have a soy allergy, then don't drink it). Also, the nutritional profile in terms of vitamins and minerals is different from cow milk (much less calcium, no B12), unless fortified to match. Soy protein is generally not considered as "complete" as cow milk protein, though that is unlikely to be of much significance if you are not training at elite levels and otherwise have a healthy diet with other protein sources in addition.