In some of the low-end "gaming machine" buildouts on sites like tomshardware, they'll often spend half the price of the machine on the graphics card.
That's because the video card is the most important piece of hardware when it comes to playing games.
Gaming Priority should usually be:
Video Card (GPU & Memory on board)
Input Devices (Mice, keyboards, etc.)
Stay away from laptops and all-in-ones if you can. You'll get more bang for your buck with a desktop system.
You forgot the Motherboard.
The Motherboard (Mainboard) is 100% the most important piece of hardware in your computer; it determines everything else that goes into your computer.
Thankfully, MOST Mainboards are fairly cheap, no more than $150-200 (usually $75-125 for a decent one) and the difference between them isn't as drastic in terms of performance.
The difference, however, is mostly what the mainboard can support
. You always, always, always start with the Mainboard first when building your computer. Once you know what your mainboard can do, then you start buying the CPU, RAM, and Graphics.
It is the Mainboard where pre-built computers sold at stores get skimped, too -- they give you unbelievably cheap prices for computers. They all use proprietary mainboards (and cases, and sometimes PSUs) that work with only their computers. The mainboards in these pre-builts are oftentimes inferior to ones you could buy from Newegg.