Butler-Sloss abuse inquiry 'would have led to Kincora'
He said the inquiry focused only on social workers and staff, excluding visitors to the east Belfast home.
Three senior staff at Kincora were jailed in 1981 for abusing 11 boys.
Lady Butler-Sloss stood down on Monday as chair of the newly-established child abuse inquiry that was set up to examine how state institutions handled their duty of care to protect children from paedophiles.
She had been under pressure to quit from MPs and victims concerned about her family links.
There have been calls for the inquiry's remit to be extended to include activities at Kincora Boys' Home in east Belfast.
Mr Goldsmith told the BBC's World at One programme that the Kincora inquiry's terms of reference "were rewritten just before it began with a view to excluding investigations into visitors to the care home so it would only focus on social workers or the staff".
He added: "These kinds of things are really big and it's inevitable that a proper all-encompassing inquiry would find its way all the way to Kincora.
"It would look at who set the term of reference, it would look at who was excluded, who was protected by the terms of reference, and that would lead to Havers himself who was responsible for that."
Amnesty International last week called for the remit of the Westminster inquiry to be widened to include Kincora.
"Allegations have persisted that paedophilia at Kincora was linked to British intelligence services, with claims that visitors to the home included members of the military, politicians and civil servants, and that police investigations into abuse at Kincora were blocked by the Ministry of Defence and MI5," said its Northern Ireland director Patrick Corrigan.