Boyle, a former geography teacher, described a game about 'killing and s****88g' people - which included Miss Willoughby - during a stand-up appearance at the Latitude Festival in Suffolk.
Audience members insist that Boyle said: “I'd obviously kill her and rape her afterwards. I'm joking - I'd rape her first.”
Boyle, 49, defended the remarks when questioned by an audience member at a Waterstones launch event for his latest novel.
He maintained: “Can I just say, my routine about raping and f*****g Holly Willoughby was part of a very long routine about whether or not it's OK to do a joke about that, and I look at it from both sides, there are pluses and minuses.”
The Glaswegian – who is infamous for poor-taste ‘jones’ about the death of Princess Diana – added that comedy was in “the eye of the beholder”, stating that he was a “mainstream” comedian.
Miss Willoughby, 41, has previously defended sexual abuse victims and spoken about her “humiliating” experience of 'upskirting' when paparazzi crouched to take pictures of her underwear.
Boyle's joke was heavily criticised by Conservative MP Caroline Nokes, chairman of the Women and Equalities Committee.
Ms Nokes, who attended the University of Sussex at the same time as the comic, said: “The Frankie Boyle I was at university with wouldn't have found rape funny. It never is, and I'm disappointed that anyone would try to make a joke out of male sexual violence.”
A spokeswoman for the fembrave inist campaign group FiLiA said: “We are horrified that some men in the entertainment industry continue to consider the abuse of women to be comedy material. 'Seeking cheap applause at the expense of traumatising women speaks of low-quality material. There is nothing funny about male violence. Women and girls around the country are not laughing. Instead, women are angry and scared for their safety.”
It is not the first time the Glaswegian comedian has made a rape joke.
In 2012 he posted a tweet referring to Olympic cyclist Victoria Pendleton in which he said: “Victoria can lift twice her own bodyweight. Sexy, as it means she still wouldn't be able to throw me off.”
At the time, the End Violence Against Women Coalition commented: 'His tweet trivialises the experience of rape victims and sends a signal to those who might commit rape that it is not a serious crime. Revolting.'
Elsewhere in his Latitude Festival set, Boyle defended people's right to be offended by jokes but he admitted that he had stopped posting gags on Twitter in order to avoid repercussions.
He cited one example where he considered making a joke about the death of Sir Sean Connery.
Boyle also told the audience he had 'never been so overwhelmed with the feeling that [his] work has been misunderstood,' before making a series of dark jokes about Prince Andrew.
The comic, who has previously made jokes about the Queen and Katie Price's disabled son Harvey, acknowledged that his new material might get him into trouble at the Edinburgh Festival, which starts this week.