He is the elder son of the publisher Sir Rupert Hart-Davis (1907 - 1999), and hence the elder brother of the television broadcaster and author Dr. Adam Hart-Davis. His sister is married to the present Lord Silsoe.
His godfather was the adventurer and writer Peter Fleming (1907 ? 1971), whose life was written up by Duff Hart-Davis as Peter Fleming: A Biography, published in 1974.