The man doesn't get to vote on whether she aborts her baby or not.
He does if he just voted to make it illegal for her to. That's what I mean at that "basic level". If only a woman's opinion matters on whether she gets the abortion, how can a man's opinion matter of whether or not it's even legal for her to do so? Sure, if the man votes yes on abortion legality, the woman still has a choice. But if the man votes no on abortion legality, that takes away her choice. There are more male congress critters than female.
A man's opinion shouldn't matter in deciding about an individuals decision to have a baby or not. Another woman's shouldn't either.
The courts have decided to make it a legal issue though. As long as it is, we the people - man or woman, get to vote or elect politicians to vote for us to make laws that we agree with.
I get to vote my opinion on things that don't directly impact me all the time.
If, back in the mid 1800's only black people were allowed to have an opinion and/or vote about slavery in the US because only black people were slaves, we'd probably still have slaves.