This also means the merits in Avatar Physical Attack are completely and utterly worthless. With 5 merits you'd have Attack+10, equivalent to 2% increase in attack.
2% is not completely and utterly worthless. Completely and utterly worthless is 0%. In this game, 2% is actually fairly significant. People pay millions upon millions for 1% more haste, for 1% more accuracy, or for 1% more MP efficiency. I love my physical attack merits.
It's all relative, really. Magic Acc merits are a complete joke, magic BPs almost never resist. Physical Acc merits are similar... it's possible to get so much accuracy now through skill and equipment that it's hard to imagine it won't be capped on BPs, so the only real benefit for Phys Acc merits is for the normal melee swings of the avatar. I honestly think the two Acc categories have the least payoff, and the two Attack ones have the most.
However, since I basically never use magic BPs at all, I have yet to drop my Phys Acc merits in favor of Magic Atk merits. For now, I'm quite content at full physical, and my SMN seems to utterly destroy every other SMN put up against it. It's often joked that the equipment and merits available for SMN don't even really affect the damage, it's just as random with or without it. I found this to be true for a long time, but then at some point I pulled ahead of the crowd. I don't understand exactly why my Pred Claws is so much more powerful than the other SMNs I play with, but I'm afraid to touch it now that it's working so well.
I think in the end, what's better (magical or physical) may really come down to what's more practical for you. For magical to excel, you really need all the bells and whistles. Get your Pet:MAB+11 staff, pick up an Eidolon Pendant, get your Caller's Pendant & Sash, etc. Merit Heavenly Strike 5/5 and magic attack 5/5, and of course, get Atma of the Beyond. For physical to excel, you need your Summoner's Doublet, Bracers, and Pigaches, and use your Sacrifice Torque as more than just an anti-sleep device. Merit 8/8 summoning skill, along with physical attack and possibly physical accuracy. If trials and newer content isn't very accessible to you, physical is much easier to pull off on the older equipment. If older stuff like relic armor isn't accessible to you, then you may wish to focus on the magic aspect more. If you're somewhere in between, you get to pick or do both!