There isn't a thing called dairy intolerance you are either intolerant to the sugar (lactose) or to the proteins in milk, cream and products produced from them.
The intolerances are due to not having enough of or any at all of the necessary enzymes to breakdown either the sugar or proteins in milk, cream and products produced from them. So this means the amount of intolerance to the sugar or proteins in milk vary person to person.
So while some people are fine with milk and related products from other mammals others aren't. This is due to different milks containing different amounts of lactose and having proteins with different structures.
In my case milk from any mammal is out there as I have a friend who can have milk sometimes but not in large qualities. On the other hand I can cope with lactose in tablets due to the small amount that is in the majority of them, and eat small amounts of cheese but know someone else who cannot cope with this at all.
Talking to a couple of people who have issues with milk they actually blame eating too much processed food as lactose and casein are used as fillers or binders in lots of processed food. In my case I suspect my lactose intolerance is inherited as my dad never drank milk or ate dairy products.