Besides that, at least for myself I don't use milk as a bevy, I use 2% milk as a condiment for coffee and tea, and occasionally on cereal (like 2-3 times a month)
While cutting out milk might be the right decision for some people (depending on how much they consume/their tolerance etc) for many/most people it wouldn't make much if any difference, I know it wouldn't for me.
I consume far more calories from eating sweets than having milk in my coffee - so it makes way more sense for me to cut down those completely empty calories than to cut out something that isn't, in moderate quantities, unhealthy, and that brings me a lot of joy. (I cut sugar out of my coffee but cutting out milk is a non-starter.)
Trolling aside, everything in moderation is really the point. If you have 2oz of a milk a day in your coffee, that's minimal and is definitely not what is making you fat. But when you factor in the "sweets" (pastries? butter? cream?) that's more cow fat. Then you factor in all the eggs. That's chicken fat. Then on top of all that, the empty sugar calories, and the fatty meats, all that sh*t adds up. Quibbling over milk in your coffee or the occasional bowl of cereal is moot. I'm saying, from my experience, even pre-veganism, I found soy milk/rice milk/almond milk to be a very good substitute in many cereals, and they make soy/coconut/almond creamer to put in coffee, which is definitely a decent substitute.
It's trying those substitutes and finding things that work for you that help you cut out calories that you have an emotional dependance on, because you can create that same emotional connection with something that tastes almost just the same, without slowly killing yourself in the process.