The Industrial Revolution marks the most fundamental transformation of human life in the history of the world recorded in written documents. For a brief period it coincided with the history of a single country, Great Britain. An entire world economy was thus built on, or rather around, Britain, and this country therefore temporarily rose to a position of global influence and power unparalleled by any state of its relative size before or since, and unlikely to be paralleled by any state in the foreseeable future. There was a moment in the world's history when Britain can be described, if we are not too pedantic, as its only workshop, its only massive importer and exporter, its only carrier, its only imperialist, almost its only foreign investor; and for that reason its only naval power and the only one which had a genuine world policy. Much of this monopoly was simply due to the loneliness of the pioneer, monarch of all he surveys because of the absence of any other surveyors. When other countries industrialized, it ended automatically, though the apparatus of world economic transfers constructed by, and in terms of, Britain remained indispensable to the rest of the world for a while longer. Nevertheless, for most of the world the 'British' era of industrialization was merely a phase - the initial, or an early phase of contemporary history. For Britain it was obviously much more than this. We have been profoundly marked by the experience of our economic and social pioneering and remain marked by it to this day.

In Germany there wasn't any alternative left. Liberalism was failing. If I'd been German and not a Jew, I could see I might have become a Nazi, a German nationalist. I could see how they'd become passionate about saving the nation. It was a time when you didn't believe there was a future unless the world was fundamentally transformed.

Tristram Hunt: Martin Amis's new book, Koba The Dread, has impugned the British Left - and you personally - for not condemning Stalin's atrocities. In your autobiography you vividly bring out the mindset of a believing Communist in the 1940s and 1950s: the party discipline and a reluctance "to believe the few who told us what they knew" of Soviet Russia. Yet you also bring out the historical context for joining the Communist Party - the battle against fascism on the streets of 1930s Berlin and a strong sense of the idealism of the October Revolution. There also remains the broader historical context that the Soviet Union remained a viable economic and political model to many in the West right up to the 1970s. Do you think this historical context seems absent in the current debate about "Communist guilt"?

Eric Hobsbawn: I must leave the discussion of Amis's views on Stalin to others. I wasn't a Stalinist. I criticised Stalin and I cannot conceive how what I've written can be regarded as a defence of Stalin. But as someone who was a loyal Party member for two decades before 1956 and therefore silent about a number of things about which it's reasonable not to be silent - things I knew or suspected in the USSR - I don't want to be critical of a book which brings out some of the horrors of Stalin. It isn't an original or important book. It brings nothing that we haven't known except perhaps about his personal relations with his father. But I don't want to say anything that might suggest to people that I'm in some ways trying to defend the record of something which is indefensible.

Tristram Hunt: Amis has criticised those on the Left who deny any moral equivalence between Nazism and Communism because the latter committed atrocities in the cause of a higher social ideal as opposed to racial genocide. The majority of deaths in the Soviet Union came not from political or racial persecution but famine caused by economic policies. As you have written of Stalin: 'His terrifying career makes no sense except as a stubborn, unbroken pursuit of that utopian aim of a communist society.' I want to tease out this issue of idealism. You stayed in the party after 1956 partly because of solidarity to the fallen and partly because of a belief in a societal ideal. Are you still drawn to an Enlightenment ideal of societal perfectibility or have you come to accept the limits of the human condition - what your friend Isaiah Berlin called, 'the crooked timber of humanity'?

Eric Hobsbawn: Why I stayed in the Communist Party is not a political question about communism, it's a one-off biographical question. It wasn't out of idealisation of the October Revolution. I'm not an idealiser. One should not delude oneself about the people or things one cares most about in one's life. Communism is one of these things and I've done my best not to delude myself about it even though I was loyal to it and to its memory. The phenomenon of communism and the passion it aroused is specific to the twentieth century. It was a combination of the great hopes which were brought with progress and the belief in human improvement during the nineteenth century along with the discovery that the bourgeois society in which we live (however great and successful) did not work and at certain stages looked as though it was on the verge of collapse. And it did collapse and generated awful nightmares.

Tristram Hunt: What struck me in your autobiography was that despite your lifelong Communist Party membership, you were deeply hostile to Militant Tendency attempts to take over the Labour Party during the 1980s. Indeed, to the fury of your comrades you became a committed supporter of Neil Kinnock's modernisation of the party - describing the 1992 general election night as the 'saddest and most desperate in my political experience'. Yet you have spoken out against Tony Blair, branding him "Thatcher in trousers". Surely New Labour was the inevitable conclusion of Kinnock's modernisation process?

Eric Hobsbawn: Most communists and indeed most socialists disagreed at the time (1980s) with the few of us who said it's absolutely no use, the Labour Party has got to go in a different direction. On the other hand, what we thought of was a reformed Labour Party not a simple rejection of everything that Labour had stood for. Obviously, any Labour Government, however watered down, is better than the right-wing alternative as the USA demonstrates. But I'm not absolutely certain that Labour Prime Ministers who glory in trying to be warlords - subordinate warlords particularly - are a thing that I can stick and it certainly sticks in my gullet.

"I've never tried to diminish the appalling things that happened in Russia, though the sheer extent of the massacres we didn't realise," says Hobsbawn. "In the early days we knew a new world was being born amid blood and tears and horror: revolution, civil war, famine - we knew of the Volga famine of the early '20s, if not the early '30s. Thanks to the breakdown of the west, we had the illusion that even this brutal, experimental, system was going to work better than the west. It was that or nothing."

He says of Stalin's Russia: "These sacrifices were excessive; this should not have happened. In retrospect the project was doomed to failure, though it took a long time to realise this." Yet he appears to argue that some goals are worth any sacrifice. "I lived through the first world war, when 10 million-to 20 million people were killed. At the time, the British, French and Germans believed it was necessary. We disagree. In the second world war, 50 million died. Was the sacrifice worthwhile? I frankly cannot face the idea that it was not. I can't say it would have been better if the world was run by Adolph Hitler."

MI5 amassed hundreds of records on Eric Hobsbawm and Christopher Hill, two of Britain’s leading historians who were both once members of the Communist party, secret files have revealed.

The scholars were subjected to persistent surveillance for decades as MI5 and police special branch officers tapped and recorded their telephone calls, intercepted their private correspondence and monitored their contacts, the files show. Some of the surveillance gave MI5 more details about their targets’ personal lives than any threat to national security.

The files, released at the National Archives on Friday, reveal the extent to which MI5, including its most senior officers, secretly kept tabs on the personal and professional activities of communists and suspected communists, a task it began before the cold war. The papers also show that MI5 opened personal files on the popular Oxford historian AJP Taylor, the writer Iris Murdoch, and the moral philosopher Mary Warnock after they and Hill signed a letter supporting a march against the nuclear bomb in 1959....

Hobsbawm, who was refused access to his files when he asked to see them five years ago, died in 2012, and Hill died in 2009. Many passages, sometimes whole pages, of their files remain redacted and an entire file on Hobsbawm has been “temporarily retained”. The files include long lists of names and addresses of letters written by Hobsbawm and Hill.

They make clear that MI5 frequently read – or was sent – copies of as many as 10 letters a day. At the same time, its officers, or special branch officers, or their informants – one of whom was given the codename Ratcatcher – were secretly taking notes of their phone calls and meetings.

The files show that Hobsbawm, who became one of Britain’s most respected historians and was made a Companion of Honour by Tony Blair, first came to the notice of MI5 in 1942 when he and 38 colleagues were described as being “obvious members of the CPGB [the Communist party of Great Britain] on Merseyside”. He became number 211,764 on MI5’s index of personal files. Although he was cleared of “suspicion of engaging in subversive activities or propaganda in the army”, MI5 noted it was doubtful that he would be suitable for the Intelligence Corps. Roger Hollis, later head of MI5, and Valentine Vivian, the deputy chief of MI6, prevented him from joining the Foreign Office’s political intelligence department.

At the end of the war, in July 1945, an MI5 officer noted: “As he is known to be in contact with communists I should be interested to see all his personal correspondence”.

MI5 said the object of keeping checks on Hobsbawm was “to establish the identities of his contacts and to unearth overt or covert intellectual Communists who may be unknown to us”. Similarly, Hill was kept under surveillance, the files note, to establish “the identity of his contacts at the University (of Oxford) and in the cultural field generally, and to obtain the names of intellectuals sympathetic to the (Communist) party who may not already be known to us”.

One early file on Hobsbawm describes his uncle Harry, with whom he sometimes stayed, as “sneering, half Jew in appearance, having a long nose”.

The surveillance intruded into the targets’ relationships. Hobsbawm is recorded in 1952 as having “difficulties with his (first) wife, who,” an MI5 officer noted, “does not consider him to be a fervent enough Communist”.

At a fraught meeting at the party’s headquarters at King Street in London’s Covent Garden, at the end of 1956, Hobsbawm, Hill and the writer Doris Lessing agreed to write a letter attacking the party leadership’s “uncritical support … to Soviet action in Hungary”, a reference to the crushing of the uprising there. That support, the letter explained, was “the undesirable culmination of years of distortion of facts”. Hill, who left the party a year later, used the phrase “the crimes of Stalin” at the meeting, according to the MI5 report. The party’s paper, the Daily Worker, refused to publish the letter which was later run by Tribune, the leftwing weekly.

Unlike the very public manifestation of McCarthyism in the US, the discreet British version had its victims. Although political activities did not affect Hill’s academic career, Hobsbawm was prevented from getting the Cambridge lectureship he wanted. He was later appointed professor at Birkbeck College, London.

The documents show that years later MI5 was furious with the BBC for allowing Hobsbawm to broadcast. In October 1962, an MI5 officer noted: “My BBC contact tells me that Hobsbawm is still an occasional contributor to the Third Programme … Some recent talks were entitled ‘Sicilian Peasant Risings’ and ‘Robin Hood’.” What is described as “slightly unexpected” was a series of talks on “Jazz”.

Earlier that year, MI6 asked MI5 if they had any objection to telling the CIA that Hobsbawm was going on a tour of South America funded, to its surprise, by the Rockefeller Foundation (Hobsbawm had already visited Cuba). In a document marked Top Secret, dated 13 May 1963, MI5 told MI6: “A reliable and very delicate source has reported that Hobsbawm visited a number of countries.”