Ant & Dec tell all in their new book Once Upon A Tyne.
In the chapter which focuses on their experiences of Britain's Got Talent, the duo revealed that Simon eyed them up as potential judges for the 2012 series, after David Hasselhoff and Michael McIntyre left the show.
Writing in their new book, they say: "David and Michael only stayed more one series, which meant that a year later the show was looking for new judges to join Amanda on the panel. Obviously, they spoke to a lot of different people about the roles, and knowing how things work, I imagine they had a lot of people in the mix at the same time. But there were two individuals they approached who we though were a very, very peculiar choice to be the new judges on BGT... Us."
"It happened in LA, where we'd stopped off on the way to Australia for I'm A Celebrity, and got a message to say Simon wanted to speak to us about the next series. Before we met him, we were expecting him to say he'd had some ideas for the judging panel and run them past us. And, in way, that's exactly what happened."
Ant continued: "He told us he had a great idea- he thought Dec and I should join the judging panel."
Contributing to the book, Simon explained: "They absolutely hated it. And, of course, I loved it because they hated it so much. But at the same time, I couldn't do what they do. I could not talk to these contestants after they've done a terrible audition and pick up the pieces. I think that's much tougher than what we do."
Dec: "Can you imagine? Simon at the side of the stage, talking to acts after they'd been judged by us two..."
Ant continued: "We talked about Simon's offer on the flight over to Australia, then slept on it - the idea, not the flight, although we did also sleep on the plane; it's a long journey and they give you special pyjamas."
"But, after much deliberation, we decided against becoming judges. We though we belonged firmly on the side of the acts, at the side of the stage, and that us joining the panel would bring unwanted pressure. Plus, we thought that because we know how hard all the acts work, we'd want to put every single on of them through!"
Ant & Dec: Once Upon A Tyne is available now.