And elemental attributes DO affect attack and defense of elements, it's in the help manual at the aetheryte. They also affect shard/crystal drops when gathering(from a newer dev Q&A), so that's something to think about.
Thank you NinjaRensha for pointing that out, as I was thinking it needed to be done before I got to your post in the thread. The in-game manual "Call to Adventure" does in fact indicate that elemental attributes affect both elemental defense and elemental spell potency. The "Ask the Devs" topics on the Lodestone are also a very good source of information to get in the habit of checking regularly.
I also wanted to share some elemental polarity (astral or umbral) theories I've come across. There was a really good debate going on over at Eorzeapedia that I thought was interesting: http://www.eorzeapedia.com/forums/viewtopic.php?f=53&t=7316
It seems that most players believe each element to potentially have an Astral polarity, an Umbral polarity, or both. Some players believe that the 3 Conquest Elements (Earth, Lightning, and Water) have an Astral polarity, while the 3 Submission Elements (Fire, Ice, and Wind) have an Umbral polarity. Others believe that each element has both an Astral and Umbral side to them due to the order of the Eorzean calendar. Another theory is that the 3 elements on the upper half of the elemental wheel (Wind, Lightning, and Fire) are Astral, with the 3 elements on the lower half (Water, Ice, and Earth) are Umbral.
As for me, I am allocating my points according to the spells I use for Conjurer. I don't see any point in hanging on to them until all these theories are tested in full, plus you can always reassign them later.