John Swinney said he offered to leave Government in 2016 and 2021, and that he also came "incredibly close to resigning" over the controversial system of exam moderation in 2020 during the pandemic.
He and the outgoing First Minister were speaking to the BBC's Nicola Sturgeon Podcast.
Ms Sturgeon spoke of the pair's close friendship, saying: "The most important person in my adult life, outside my husband and family, is actually John Swinney."
Mr Swinney was Scottish education secretary at the onset of the pandemic, when exams were cancelled and a new grading system was put in place in which teachers estimated pupils' attainment and it was then moderated by the Scottish Qualifications Authority (SQA).
But that resulted in 124,564 pupils having their results lowered from the teacher estimates. Following an outcry, Mr Swinney apologised and told the SQA to revert downgraded results to the estimates.
Mr Swinney said: "I came incredibly close to resigning. Very, very close. Because I felt I had made a mistake. What just about persuaded me to do it was the sense that emerged that Covid had turned everything upside down, but we were trying to say 'business as usual' on certification.
"I remember reading a comment which said: 'Surely the kids of Scotland could have been given a break because everything else has been turned upside down'.
"I read that comment and thought, 'that's your mistake mate, and you should own it'."
He later survived a vote of no confidence at Holyrood over the moderation system in an opposition bid to oust him.
Mr Swinney said he offered to stand down from the Cabinet after the Scottish Parliament elections in 2016 and 2021, but said of Ms Sturgeon: "She basically wouldn't countenance me leaving government."
Before Christmas 2022, Mr Swinney said he told Ms Sturgeon of his intention to quit government office once his role of acting Finance Secretary ended when Kate Forbes returned from maternity leave.
He said on that occasion, Ms Sturgeon did not try to stop him.
The First Minister admitted she may have already been thinking about her own future at that point.
Ms Sturgeon said: "If you said to me then that come the middle of February you are going to be announcing your resignation, I would have said to you, 'Don't be ridiculous, no I won't be.'
"But looking back on it, I think subconsciously I was already grappling with that, and maybe in my response to John there was a sense of what deep down inside maybe I knew what was coming for me as well."