An RAC poll of 2,498 drivers indicated that 46% think it should take no longer than that to get going when the lights change.
The motoring organisation expressed concern over the impact on road safety, traffic jams and air pollution from drivers being too slow to react.
Some 17% of respondents to the survey said they regularly fail to get through lights due to the inaction of drivers in front.
Nearly two-thirds (64%) claim to have been left waiting on green for more than 10 seconds.
Men are generally more likely than women to be irritated by drivers who are slow to start moving (50% compared with 41%), the survey suggested.
Motorists from Yorkshire and the Humber region are the most impatient, with 55% saying they get annoyed by the time it takes others to pull away.
Drivers in Wales are the most tolerant, with just 34% frustrated.
Around a third (35%) of respondents said between four and six seconds to move off is reasonable, while a relaxed 7% are prepared for other drivers to take as long as they like.
RAC spokesman Simon Williams said anything longer than three seconds “can start to seem like an eternity when you desperately want to get through a set of traffic lights”, particularly as some only stay green for 15 seconds.
He went on: “This severely limits the number of vehicles that can get through before red comes up again, and in turn makes jams – and potentially even air pollution – worse.
“Too many drivers – for whatever reasons – clearly aren’t paying enough attention to what’s going around them when they’re stationary at traffic lights.” Mr Williams added.
“As well as increasing journey times, this could have a road safety implication, particularly when people suddenly realise the lights have changed and then hurriedly pull away without properly checking their mirrors and making sure everything is clear around them.
“More drivers need to pay attention to the lights and not use the time to daydream or worse to check their handheld phones illegally.”