I'm usually pretty careful to keep my hands warm. Once your hands go numb, you've got real problems if you need to rope up, put on crampons, or just zip up your jacket.
But a couple weeks ago I set out for a small, easy peak. I knew the weather was cold (around ten F) and there'd be a breeze at the top (gusts to 30mph or so) but I was trying to move fast so I wore no insulation on my legs, just a softshell (which isn't windproof), and I soon stripped down to a light fleece. That was good as long as I kept chugging uphill, but I was also running a little late so I never stopped for food or water. By the time I hit the top and suddenly was exposed to the wind, I was pretty spent. Luckily I'd put on a couple layers (including my down jacket) just before hitting the top, but by then my hands had already started going numb and they didn't fully recover until a while later when I was on my way down. Pretty stupid, really.
Even stupider was what I did earlier in the season. I was looking to go ice climbing at an unfamiliar crag. I was wearing snow pants, fleece, and belay jacket, and carrying a pack with my rope, screws, etc. For some reason I wasn't wearing gloves. So I started uphill on a rough climber's trail, looking for decent ice, and of course I needed to use my hands, which meant plunging them into snow to look for handholds. After about ten minutes of this I almost passed out - from overheating. Even though my hands were painfully cold, I was wearing far too many layers for the level of exertion. After two minutes of sitting in the snow with my jacket unzipped, I was fine - though my hands were still cold.
The moral is: you need to monitor your extremities separately from your core. Letting either one get too cold or too hot can be a problem.
My feet almost never get cold, but I find I have to pay attention to my hands - they go quickly from cold to sweating, and back.