From a logical stand point, infants are at more risk, for the simple fact that they can not voice that they are cold, whereas that is sometimes the ONLY complaint the elderly voice (and voice often...)
Now when you start talking about dementia and Alzheimer's, and other degenerative conditions of the brain, I would put them at about the same level as infants, or even potentially more at risk, because the general instinct is to cover and wrap infants (too much sometimes), whereas the elderly are often left to their own devices. It is not unlikely, in fact maybe even a commonality, to find an elderly person with dementia who has wandered out into the cold without a coat, or falls on the way to the mailbox, only to be discovered hours later.
Just from a patho stand point though, infants have a higher risk for hypothermia. As has already been pointed out, they have a higher BSA to mass ration, and lose heat almost insensibly when left uncovered. The nervous systems of infants are less developed too, making their ability to shiver or use other muscular methods of generating heat minimal.
I have been to referrals to transfer infants who have been left exposed on a hospital bed or in an ohio warmer that was not turned on, and witnessed core temps of 95 or 96 before. It is not that hard to do. Keep those babies covered! Unless they have a fever, in which case, a normal layer of clothing appropriate for the season is good enough.