The Defence of the Realm: A History of MI5

LSE Public Lecture, October 15, 2009
Speaker: Professor Christopher Andrew
Host/Chair: LSE Film and Audio Team (Arne Westad and colleagues)

Overview

This episode features Professor Christopher Andrew discussing his landmark authorized history of MI5, "The Defence of the Realm." In a lively, anecdote-rich lecture followed by a probing Q&A, Andrew journeys through MI5’s 100-year history—from its creation in 1909 to its modern focus on counterterrorism—unpacking what really happens inside Britain's storied domestic intelligence service. He explores MI5’s role in the World Wars, the Cold War, its surprising history regarding gender and recruitment, and its evolving response to new threats like Islamist terrorism. Throughout, Andrew offers frank reflections on the myths, successes, and failures of MI5, and the ongoing importance of understanding intelligence in British—and international—history.

Main Themes

• The origins, functions, and evolution of MI5
• Wartime espionage, double agents, and the roots of British deception operations
• Myths and realities around MI5’s reputation (e.g., the "Reds under the bed" era and James Bond culture)
• Gender history within MI5 and early breakthroughs for women
• MI5’s changing priorities, especially the shift to counterterrorism
• The enduring value—and underappreciation—of intelligence history in broader historical analysis

Key Discussion Points & Insights

1. MI5 Recruitment, Myths, and Identity

[02:43-08:30]

• Recruitment Cultures: Cambridge’s quirky history of recruiting both spies for Britain and, infamously, Soviet agents; contrast with LSE's "conventional" approach.
• The “Cambridge Five”: The KGB’s top recruits all crossed paths at University College London, debunking dominant Cambridge-only myths.
• James Bond Fiction vs. Reality: MI5 is more bureaucratic than glamorous, but featured some daring real-life agents—like Christopher Draper (the “Mad Major”) who wowed both British and Nazi audiences flying under London bridges.

Quotes:

“Anyone who writes about intelligence is acutely aware that it’s the only profession in which a fictional character is at least 100 times better known than anyone who actually worked for that profession at any time in the past or present.”
—Christopher Andrew, [06:11]

2. Double Agents, Deception, and the Art of Espionage

[09:00-19:10]

• The Origins of British Espionage Success: In WWI, MI5's early triumphs included rounding up German spies, which may have saved France from a six-week defeat.
• Key figures such as William Hinchley Cook, whose bilingual skills enabled elaborate undercover work.
• Double-Cross System: The legendary deception campaign against the Nazis in WWII traced roots back to unlikely beginnings (e.g., a pilot’s publicity stunt).

Quotes:

“It is through following Draper’s contacts with German intelligence... that MI5 learns which addresses in Germany to monitor... It is able to turn the best of them around into the so-called double cross system, which I think few nowadays challenge as the most successful deception in the entire history of warfare.”
—Christopher Andrew, [04:53]

3. MI5’s Self-Image, Bureaucracy, and Gender Dynamics

[19:11-28:00]

• The Christmas Card Analogy: MI5’s own illustrated holiday card reveals its 1920s self-image—vigilant, patriotic, but reactionary in its view of “subversion”.
• Breaking the Gender ‘Glass Ceiling’: MI5 was pioneering for employing women in responsible roles from WWI (e.g., chief financial planning), long before other government bodies or companies.
• Playfully highlighting mixture of innovation and dated gender attitudes in early MI5, including flirtatious office culture and secretarial advancement.

Quotes:

“The very first place in Britain in which the glass ceiling was broken by women was actually in MI5...”
—Christopher Andrew, [23:29]

4. Eccentricity, Creativity, and Influence

[28:01-35:00]

• Colorful Characters: Maxwell Knight, MI5’s “agent runner” moonlighted as a BBC naturalist, kept a parrot, and recruited agents from across the ideological spectrum, exemplifying the eccentric but effective agent-manager stereotype.
• The Power of Ridicule and Creativity: Deceptive communications to German intelligence during WWII often used humor to creative effect.

Quotes:

“You do not embark on a period of deception like [the double cross system] unless you have learned from a previous track record.”
—Christopher Andrew, [06:55]

5. Postwar Shifts: The Cold War, Subversion Fears & Modern Terrorism

[35:01-45:00]

• Cold War Myths and Realities:
  • Contradicts popular belief that MI5 was obsessed with Red subversion—actually, government ministers drove much of the paranoia.
  • MI5 sometimes found Communist Party plots more bumbling than formidable, providing reassurance rather than stoking panic.
• Counterterrorism:
  • After WWII, MI5’s main focus shifted through the decades: Jewish militant groups (Irgun), later the IRA, and eventually Islamist terrorism.
  • MI5’s delay in grasping the full dimension of Islamist-inspired threats (e.g., deferred recognition of the seriousness of al-Qaeda cells pre-9/11).

Quotes:

“The idea that weapons of mass destruction will be the only invention in the entire history of the human race not to spread around the world is a proposition so improbable that I really find difficulty in taking it in.”
—Christopher Andrew, [55:46]

6. Failures, Learning, and The Importance of Historical Perspective

[45:01-57:00]

• Failures and Oversight: Examples include slow adaptation to the IRA threat in the UK due to unclear division of lead agency roles; later, similar issues in responding to Islamist terrorism.
• Role of Historical Hindsight: Defends the use of “historian’s benefit of hindsight” as vital, not a flaw.
• Critiques policymakers and intelligence agencies for neglecting history, advocating long-term perspective as the essential key to agility and effective strategy.

Quotes:

“Any historian who does not use historical hindsight should be sacked immediately.”
—Christopher Andrew, [54:41]
“The secret of long-term planning is to accept the inevitability of surprise... Take the long-term view—and expect to be surprised.”
—Christopher Andrew, [76:09]

7. Q&A Highlights

[47:04–end]

Notable Audience Questions:

• Surprises in the Archive:
  • Biggest surprise was the frequency with which MI5 found the Communist Party more incompetent than threatening ([48:06]).
• On Constraints of Authorized History:
  • Andrew insists he maintained independence, and since MI5 never anticipated these files going public, attempts at sanitization were rare ([54:21], [71:08]).
• The Harold Wilson "Plot":
  • There is no credible evidence MI5 plotted against Labour PM Harold Wilson; conspiracy theories about this are unfounded ([58:52]).
• Response to Islamist Terrorism:
  • MI5 was late in recognizing the seriousness, but “so far as we can see... it’s done pretty well.” Points to Operation Overt as a prevented "British 9/11" ([67:00]).
• Archive Completeness and Doctoring:
  • Extensive records, little evidence of revisionism; archives can’t be easily “doctored” due to process controls ([72:00]).
• Agility vs. Long-Term Planning:
  • Effective intelligence work balances the long view with readiness to improvise in a shifting world ([76:09]).
• On Weapons of Mass Destruction and the Iraq War:
  • MI5 was not responsible for the Iraq WMD intelligence failure, as it is a domestic agency dealing with internal threats ([75:41]).

Notable Quotes with Timestamps

• On James Bond and Intelligence Myths:

  “Anyone who writes about intelligence is acutely aware that it’s the only profession in which a fictional character is at least 100 times better known than anyone who actually worked for that profession...”
  —Christopher Andrew, [06:11]

• On Eccentric Recruitment and Double Agents:

  “I’d say Cambridge has sent some of its best people into the other side as well. The KGB always regarded [the Cambridge spies] as the ablest group of foreign agents it ever recruited...”
  —Christopher Andrew, [03:03]

• On MI5’s Gender History:

  “The very first place in Britain in which the glass ceiling was broken by women was actually in MI5.”
  —Christopher Andrew, [23:29]

• On Learning from History:

  “Any historian who does not use historical hindsight should be sacked immediately.”
  —Christopher Andrew, [54:41]

• On Facing Islamist Terrorism:

  “The only way to prepare for anything really important is to take a long-term view... There are only two constants in human history over the last 20,000 years. The first one is obviously human nature.”
  —Christopher Andrew, [55:18]

• On Conspiracy Theories about Harold Wilson:

  “There is nothing I can do to help [those who believe there was a plot against Wilson].”
  —Christopher Andrew, [63:10]

Timestamps for Important Segments

• Introduction and Overview [00:00–02:43]
• MI5 Recruitment and the Cambridge Five [02:43–06:11]
• WWI Espionage and Early MI5 Success [06:12–13:20]
• Breaking Gender Barriers [19:10–24:54]
• WWII, Double-Cross System, and Agents like Juan Pujol [24:55–35:00]
• MI5’s Cold War Reputations and Subversion Fear [35:01–45:00]
• Counterterrorism and Rise of Islamist Threat [45:00–47:00]
• Q&A on Surprises, Hindsight, Authorized History, Wilson, and More [47:04–78:10]
• Closing Reflections [78:10–end]

Memorable Moments

• Eccentric Employees: Tales of a parrot-owning MI5 officer who doubled as a BBC presenter ([28:01]).
• Ridiculous—but Effective—Deceptions: The “double-cross” operation’s success using agents who sent outlandishly creative (and sometimes hilarious) misinformation ([40:00]).
• Christmas Cards as Self-Image: MI5’s self-concept crystallized in a 1920s Christmas card, complete with “masked Britannia” stabbing “subversion in the back” ([19:11]).
• Historical Insight: The thesis that MI5’s surveillance of the Communist Party mostly reassured governments of their irrelevance, rather than feeding anxiety ([48:03], [51:40]).
• Defending the Role of History: Andrew’s passionate call for more historians (and policymakers) to integrate intelligence into their analyses ([53:01]).

Conclusion

Christopher Andrew’s lecture brings MI5’s complex, colorful history to life, exposing the realities behind its public image, celebrating its eccentric and innovative characters, and challenging lingering myths. He underscores how intelligence is inextricably linked to British—and indeed global—history, and why only by shedding the "historical attention span deficit disorder" can one begin to understand the present or prepare for the future.

Listen to the episode for the full, spirited recounting and firsthand stories from the “official historian” of MI5.