Hurts the middle class the most.
The politicians will be debating oil policies forever, and often the actual facts are lost in their mumbling/persuasion/lies. The people hit hardest by the rise in oil and consequently gas prices are those with the longest commutes. Many cannot afford to live close to their work and have to drastically change their lifestyles if gas were to rise $1 a gallon.
Fuel Freedom gives us some more details on what happens when the price of oil goes up, noting that Americans spend more on transportation than clothing, healthcare, and entertainment combined.