So if I'm reading this right, there is a cap on MAB for avatars, relatively speaking. Should I not be using Ultimate then?
Sitting on the following boosts for gear (Full +2 set)
5 MAB from Soulscourge
1 MAB from Tiresias' cape
10% Boost from Caller's Doublet +2
and yet, when I went from beyond / howling / ultimate to beyond / minikin / ultimate... my damage went up from a steady 3725 to a steady 3845. Should I use Noxious Fang for a third if I want to break 4000? I'm rather confused.
There is no cap anyone has found. I am not sure what howling is, as I cannot find an atma with its name in it. However, 50 int being multiplied by a lot of MAB (ultimate, beyond, innate, and gear added together) does a lot more for your damage than you seem to think it does. MAB is not the end all be all for BLM, and such is the same for summoner. When MAB gets really high, and int is low (since avatars don't get +int gear, food, or anything else), adding INT has a much more drastic effect than MAB does.
The highest damage for grand fall should be minikin, ultimate, and noxious fang. There are four things you can toy with that are going to effect your bloodpact damage. MAB, INT, Element specific damage+ (ie noxious fang, beyond, lion, etc), and TP. Day/weather too but that's kind of separate.
Tp, especially at a level 5 bp, is something you probably won't have to play with much. And using an atma slot for Magic BP is a waste anyways.
That leaves you with MAB, INT, and Element specific damage.
Without really going into the math of it all (and since I don't know the exact math anyways) what I can tell you is that you want a great balance of them all. Gearing for just MAB (Ie beyond, ultimate, baying moon) for +110 MAB is going to do less for grand fall than you will get from minikin/ultimate/noxious fang.
Gear MAB is calculated separately then? Because it wouldn't make any sense that going from 30 MAB to 50 INT would raise my damage by 100, unless the 30 MAB was having a partial effect that added X damage where X is less than Y added by 50 INT, and therefore adding more MAB onto gear would actually boost the numbers higher. I know I could potentially get another 5 from staff and another 2 from belt, at the cost of Blood Boon... that's a trade-off since proc'ing the full +2 set bonus I imagine is quite a lot compared to a couple hundred damage on a nuke. EDIT: Also forgot Eidolon Pendant, that's another +2 that can be swapped in, for a total of 9. 9 MAB for 8 Blood Boon.
Gear isn't calculated separately. You have to keep in mind, having an innate 36 MAB, +30 from Beyond, +50 from ultimate, +9 from gear... is 125 total MAB. If you added, say another 30 MAB atma onto that, you're only increasing damage by 13.3% (2.55 / 2.25). That's not really much considering it's 30 MAB.
Adding Int onto that though... the int is getting multiplied by the 2.25 Magic attack modifier, and that is why the damage is higher. The higher your MAB gets, the more each point of int adds to your overall damage. The higher your MAB gets, the less each additional point of MAB adds to your overall damage.