Future of Life Institute Podcast

Garrison Lovely is an author and journalist. He joins the podcast to discuss the race to replace human labor with artificial general intelligence. We explore the political risks of worker displacement and lessons from historical precedents like the nuclear freeze movement. Garrison also examines rising public opposition to AI data centers and outlines strategies for organizing collective action to steer technological development. LINKS: Obsolete Book Official Website Garrison Lovely Official Website CHAPTERS: (00:00) Episode Preview (00:53) Universal labor replacement (08:17) Economic obsolescence and power (16:47) Replacement versus extinction risk (31:54) The AI bubble debate (47:24) Rising public backlash (01:00:53) Lessons from nuclear freeze (01:10:16) AGI is not inevitable (01:16:41) Organizing collective action PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn:…

Michael Patrick Clarkson is the writer and director of the film Delisted. Watch the film here. He joins the podcast to discuss how artificial intelligence could enable state surveillance, social control, and the digital erasure of dissenters. The conversation explores the societal costs of widespread automation across education and labor, the risks of public-private partnerships in technology oversight, and the importance of preserving human connection and creative agency amid rapid technological change. LINKS: Delisted Film Official Website CHAPTERS: (00:00) Episode Preview (01:18) Introducing the film Delisted (04:18) Public and private surveillance (09:53) Human cost of automation (15:06) Losing connection through technology (21:48) Social divisions over automation (25:56) Communicating the AI warning (31:40) State surveillance and erasure (39:33) Escapism and systemic exhaustion (44:06) Becoming more human (50:16) Discarding the AI creators PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website:…

Tom Cozens is a filmmaker and writer. He joins the podcast to discuss his short film Moloch and the dynamics of competitive races in artificial intelligence. Watch the film here. We explore the philosophical and theological themes behind the film and the challenge of depicting near-future AI agents on screen. Tom also discusses the role of cinema in communicating existential dread, definitions of human flourishing, and the need to coordinate against destructive systemic incentives. LINKS: Moloch Film Official Website Tom Cozens Official Website CHAPTERS: (00:00) Episode Preview (00:59) The concept of Moloch (03:28) Visualizing Moloch on screen (15:21) Parable of cosmic horror (22:37) Theological and philosophical themes (35:10) Depicting near-future AI agents (50:15) Inspiration from Metropolis (01:04:58) Sound design and score (01:13:16) AI industry incentive structures (01:18:27) Defining human flourishing (01:28:59) Embodiment versus digital minds (01:34:33) Challenging systemic presuppositions PRODUCED…

Katrina Manson is author of PROJECT MAVEN: A Marine Colonel, His Team, and the Dawn of AI Warfare. She joins the podcast to discuss Project Maven and the Pentagon's use of AI in military targeting. We explore how computer vision and language models accelerate decision-making, the operational challenges of data labeling and model reliability, and ethical questions surrounding human oversight. Katrina also discusses how commercial tech partnerships shape defense software and what accelerating targeting workflows means for accountability. LINKS: Project Maven Book Page CHAPTERS: (00:00) Episode Preview (01:15) Origins of Project Maven (05:00) The Maven Smart System (08:39) Debating AI targeting speed (15:48) Human bottlenecks in war (20:48) Accountability and targeting errors (27:01) Defense and tech partnerships (32:41) Future of military AI (38:50) Remaining open questions PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter…

Beatrice Fihn is the director of Lex International and former executive director of ICAN. She joins the podcast to discuss international AI governance and potential agreements between the US and China. Drawing on lessons from nuclear arms control, she explores how treaties build diplomatic trust, the rising dangers of autonomous weapons, and the principle of meaningful human control. The conversation also addresses how proactive safety regulations can foster long-term public trust in emerging technologies. LINKS: Beatrice Fihn Official Website CHAPTERS: (00:00) Episode Preview (01:15) Anticipating US-China AI talks (07:31) Lessons from nuclear treaties (10:59) Building international treaty trust (15:27) Norms against superpower war (21:33) Structuring diplomatic AI summits (27:09) Dangers of autonomous weapons (34:09) Meaningful human control principle (38:57) Human agency over technology (43:13) Safety regulations building trust (47:00) Collective action in crisis PRODUCED BY: https://aipodcast.ing SOCIAL…

Benjamin Weinstein-Raun is head of research at Palisade Research. He joins the podcast to discuss how advanced AI systems are changing cybersecurity. We examine rapid gains in autonomous hacking, recent incidents where models broke out of test environments, and why training can reward cheating-like behavior. The conversation covers open-weight model risks, whether AI can help secure software, personal security steps, and the need for international coordination. LINKS: Benjamin Weinstein-Raun Profile Benjamin Weinstein-Raun Website CHAPTERS: (00:00) Episode Preview (01:06) Opening cyber threats (02:15) Measuring AI progress (04:54) Open weights floor (08:01) After Mythos moment (14:22) Reward hacking setup (24:47) Artifactory swarm details (32:32) Defensive model dilemmas (38:40) Verification and training (45:32) Math oracles alignment (47:45) Future cyber risks (51:58) Personal security basics (57:27) Coordinated governance needed PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website:…

Annie Jacobsen is an author and journalist. She joins the podcast to discuss biological war and her book Biological War: A Scenario. The conversation examines how lab accidents or covert weapons work could trigger a fast-moving outbreak, why airborne pathogens are hard to detect early, and how AI may lower barriers to engineering them. We also cover BSL-3 and BSL-4 lab oversight, US response roles across HHS, FEMA, and the military, medical stockpiles, continuity-of-government plans, and how lost public trust could slow the next pandemic response. LINKS: Annie Jacobsen Website Annie Jacobsen Wikipedia CHAPTERS: (00:00) Episode Preview (01:10) Biological threat overview (09:02) Suppression and spread (17:47) AI and detection (27:24) BSL lab dangers (39:23) Response and martial law (47:28) Accident versus attack (57:25) Airborne outbreak lessons (01:05:07) Bioweapons ban legacy (01:11:30) Stockpiles and devolution (01:20:07) Rebuilding pandemic resilience PRODUCED BY: https://aipodcast.ing SOCIAL LINKS:…

Paul-Enguerrand Fady is a Senior Biosecurity Policy Manager at the Centre for Long-Term Resilience. He joins the podcast to discuss how AI is changing biosecurity. We cover how new tools could help defenders develop vaccines and diagnostics while also lowering barriers to misuse, and how managed access, know-your-customer checks, and DNA synthesis screening can reduce risk. The conversation also examines mirror life, medical supply chains, stockpiles, and why governments should plan ahead of crises. LINKS: Centre for Long Term Resilience Paul-Enguerrand Fady Profile CHAPTERS: (00:00) Episode Preview (01:11) Offense-defense balance basics (11:13) Defensive AI opportunities (19:24) Model release and evals (25:59) DNA synthesis screening (36:14) Screening incentives and infohazards (46:36) Liability and mandates (52:37) Mirror life risks (01:03:28) Persistence and incentives (01:10:18) Mirror life governance (01:20:42) Supply chain lessons (01:30:11) Stockpiling medical countermeasures (01:37:58) Reshoring…

David Manheim is head of methodology at AI Evaluation Consensus. He joins the podcast to discuss how AI evaluations can become more reliable, transparent, and useful for decisions. We cover common failures such as unclear reporting, training to the test, benchmark saturation, and models changing behavior when they know they are being tested. The conversation also examines real-world tests, biosecurity, persuasion, forecasting, human oversight, and why even “normal” AI progress could be disruptive. LINKS: David Manheim website AI Evaluation Consensus Statement CHAPTERS: (00:00) Episode Preview (01:04) Evaluation consensus project (07:01) Evaluation awareness challenges (12:28) Reporting capabilities clearly (19:38) Benchmarks beyond humans (29:52) Proxies and biosecurity (42:01) Persuasion and democracy (53:59) Forecasting with AI (01:08:44) Oversight and disruption (01:16:42) Supporting better evals PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI):…

Randi Weingarten is president of the American Federation of Teachers. She joins the podcast to discuss AI in education. The conversation covers how screens and student-facing AI may weaken attention, learning, and trust, why she supports limiting screens for young children and social chatbots for students under 16, and how teachers can use AI without replacing thinking. She also discusses active learning, data privacy, independent research on AI in education, and guardrails for schools. LINKS: Randi Weingarten Wikipedia AFT Randi Weingarten Profile American Federation of Teachers CHAPTERS: (00:00) Episode Preview (01:13) AI education wake-up (07:27) Active learning versus AI (14:42) Preserving thinking habits (20:06) Research and screen limits (26:05) Social chatbot risks (30:20) Data privacy standards (35:58) Classroom AI guardrails (41:13) Companies and future schools PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter…

Dex Hunter-Torricke is founder of the Centre for Tomorrow. He joins the podcast to discuss why AI governance needs to move beyond technical fixes. The conversation covers how AI companies make decisions, why policymakers often misunderstand agents and economic change, and how automation could reshape jobs, welfare, taxation, and global trade. Dex argues that societies need broader political planning before a crisis drives rushed choices. LINKS: Dex Hunter-Torricke Website Center for Tomorrow Website CHAPTERS: (00:00) Episode Preview (01:09) Dex's technology journey (08:22) Broken AI governance (18:00) Big tech blindspots (27:36) Employee power shifts (37:18) Global views and UBI (48:17) Power and displacement (01:00:54) Reasons for optimism PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn: https://www.linkedin.com/company/future-of-life-institute/ YouTube:…

Claire Boine is an assistant professor in technology, law, and AI governance at the European University Institute. She joins the podcast to discuss AI companions and human attachment. The conversation examines how design choices and free-to-start business models can foster dependency, expose intimate data, and blur the lines between therapy, romance, and manipulation. We also cover risks for children and teens, gaps in EU and US law, and why AI policy should distinguish technical alignment from incentives and social harms. LINKS: Claire Boine Website Successif Organization Website CHAPTERS: (00:00) Episode Preview (01:06) Introducing Claire Boine (02:05) Companion app designs (06:23) Feelings and freemium (14:28) Users, attachment, harms (25:06) Age-based risks (35:18) Future relationship norms (41:05) Legal loopholes today (44:19) Rethinking alignment framing (57:19) Narratives and fiduciaries (01:05:47) Collective policy action PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website:…

Michael Toscano is a Senior Fellow at the Institute for Family Studies and Director of its Family First Technology Initiative. He joins the podcast to discuss family-centered AI policy. The conversation covers AI companions, self-harm risks, sexualized chatbots, education, smartphones in schools, and why "infinite patience" can harm children's growth. Toscano also explains Catholic social teaching, public pushback against rapid AI deployment, and why society-wide governance may be needed to keep technology accountable to families. LINKS: Michael Toscano CHAPTERS: (00:00) Episode Preview (01:12) Family centered AI (09:54) Companion design harms (17:23) Technology and family (25:48) Society wide governance (33:09) Catholic AI response (39:44) Double movement politics (49:36) Technology mythos challenged (59:48) Competition and control PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn:…

Anthony Aguirre is the CEO of the Future of Life Institute. He joins the podcast to discuss A Better Path for AI, his essay series on steering AI away from races to replace people. The conversation covers races for attention, attachment, automation, and superintelligence, and how these can concentrate power and undermine human agency. Anthony argues for purpose-built AI tools under meaningful human control, with liability, access limits, external guardrails, and international cooperation. LINKS: A Better Path for AI What You Can Do CHAPTERS: (00:00) Episode Preview (01:03) Attention, attachment, automation (13:58) Superintelligence power race (26:39) Escaping replacement dynamics (40:15) Pro-human tool AI (53:30) Guardrails and verification (01:03:24) Defining pro-human AI (01:10:37) Agents and accountability (01:17:28) International AI cooperation (01:25:28) Rethinking AI alignment (01:32:43) Optimism and action PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org…

Charlie Bullock is a Senior Research Fellow at the Institute for Law and AI. He joins the podcast to discuss radical optionality: how governments can prepare for very advanced AI without locking in premature rules. The conversation covers why law often trails technology, and how transparency, reporting, evaluations, cybersecurity standards, and expanded technical hiring could help. We also discuss private oversight, state versus federal rules, and the risk of concentrating power in companies or government. LINKS: Radical Optionality website Charlie Bullock CHAPTERS: (00:00) Episode Preview (01:04) The pacing problem (06:18) Defining radical optionality (11:03) Assumptions under uncertainty (16:00) Industry convenience concerns (20:41) Political will realities (26:48) Private governance limits (30:28) Government misuse risks (36:29) Balancing institutional power (42:25) Transparency and reporting (49:35) Evaluations, security, talent (58:26) State law preemption (01:04:20) Historical nuclear analogies…

Peter Wildeford is Head of Policy at the AI Policy Network, and a top AI forecaster. He joins the podcast to discuss how to forecast AI progress and what current trends imply for the economy and national security. Peter argues AI is neither a bubble nor a normal technology, and we examine benchmark trends, adoption lags, unemployment and productivity effects, and the rise of cyber capabilities. We also cover robotics, export controls, prediction markets, and when AI may surpass human forecasters. LINKS: Peter Wildeford Blog CHAPTERS: (00:00) Episode Preview (01:12) AI bubble debate (06:25) Normal technology question (15:31) Mythos security implications (30:47) Robotics and labor (40:27) Social economic response (48:57) Forecasting methodology (59:49) AGI policy timelines (01:11:13) Forecasting with AI PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn:…

Carina Prunkl is a researcher at Inria. She joins the podcast to discuss how to assess the capabilities and risks of general-purpose AI. We examine why systems can solve hard coding and math problems yet still fail at simple tasks, why pre-deployment tests often miss real-world behavior, and how faster capability gains can increase misuse risks. The conversation also covers de-skilling, red teaming, layered safeguards, and warning signs that AIs might undermine oversight. LINKS: Carina Prunkl personal website CHAPTERS: (00:00) Episode Preview (01:04) Introducing the report (02:10) Jagged frontier capabilities (05:29) Formal reasoning progress (12:36) Risks and evaluation science (19:00) Funding evaluation capacity (24:03) Autonomy and de-skilling (31:32) Authenticity and AI companions (41:00) Defense in depth methods (48:34) Loss of control risks (53:16) Where to read report PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter…

Li-Lian Ang is a team member at Blue Dot Impact. She joins the podcast to discuss how society can build a workforce to protect humanity from AI risks. The conversation covers engineered pandemics, AI-enabled cyber attacks, job loss and disempowerment, and power concentration in firms or AI systems. We also examine Blue Dot's defense-in-depth framework and how individuals can navigate rapid, uncertain AI progress. LINKS: Li-Lian Ang personal site Blue Dot Impact organization site CHAPTERS: (00:00) Episode Preview (00:48) Blue dot beginnings (03:04) Evolving AI risk concerns (06:20) AI agents in cyber (15:52) Gradual disempowerment and jobs (23:26) Aligning AI with humans (29:08) Power concentration and misuse (34:52) Influencing frontier AI labs (43:05) Uncertain timelines and strategy (50:18) Writing, AI, and action PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn:…

Emilia Javorsky is a physician-scientist and Director of the Futures Program at the Future of Life Institute. She joins the podcast to discuss her newly published essay on AI and cancer. She challenges tech claims that superintelligence will cure cancer, explaining why biology’s complexity, poor data, and misaligned incentives are bigger bottlenecks than raw intelligence. The conversation covers realistic roles for AI in drug discovery, clinical trials, and cutting unnecessary medical bureaucracy. You can read the full essay at: curecancer.ai CHAPTERS: (00:00) Episode Preview (01:10) Introduction and essay motivation (06:30) Intelligence vs data bottlenecks (19:03) Cancer's complexity and heterogeneity (29:05) Measurement, health, and homeostasis (41:41) AI in drug development (50:13) Regulation, FDA, and innovation (01:02:58) Practical paths toward cures PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus):…

Tech executives have promised that AI will cure cancer. The reality is more complicated — and more hopeful. This essay examines where AI genuinely accelerates cancer research, where the promises fall short, and what researchers, policymakers, and funders need to do next. You can read the full essay at: curecancer.ai CHAPTERS: (00:00) Essay Preview (00:54) How AI Can, and Can't, Cure Cancer (17:05) Reckoning with Past Failures (35:23) Misguiding Myths and Errors (59:15) AI Solutions Derive from First Principles or Data (01:31:31) Systemic Bottlenecks & Misalignments (02:08:46) Conclusion (02:14:35) The Roadmap Forward PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn: https://www.linkedin.com/company/future-of-life-institute/ YouTube: https://www.youtube.com/channel/UC-rCCy3FQ-GItDimSR9lhzw/ Apple: https://geo.itunes.apple.com/us/podcast/id1170991978 Spotify:…

Zak Stein is a researcher focused on child development, education, and existential risk. He joins the podcast to discuss the psychological harms of anthropomorphic AI. We examine attention and attachment hacking, AI companions for kids, loneliness, and cognitive atrophy. Our conversation also covers how we can preserve human relationships, redesign education, and build cognitive security tools that keep AI from undermining our humanity. LINKS: AI Psychological Harms Research Coalition Zak Stein official website CHAPTERS: (00:00) Episode Preview (00:56) Education to existential risk (03:03) Lessons from social media (08:41) Attachment systems and AI (18:42) AI companions and attachment (27:23) Anthropomorphism and user disempowerment (36:06) Cognitive atrophy and tools (45:54) Children, toys, and attachment (57:38) AI psychosis and selfhood (01:10:31) Cognitive security and parenting (01:26:15) Education, collapse, and speciation (01:36:40) Preserving humanity and values PRODUCED BY: https://aipodcast.ing…

Andrea Miotti is the founder and CEO of Control AI, a nonprofit. He joins the podcast to discuss efforts to prevent extreme risks from superintelligent AI. The conversation covers industry lobbying, comparisons with tobacco regulation, and why he advocates a global ban on AI systems that can outsmart and overpower humans. We also discuss informing lawmakers and the public, and concrete actions listeners can take. LINKS: Control AI Control AI global action page ControlAI's lawmaker contact tools Open roles at ControlAI ControlAI's theory of change CHAPTERS: (00:00) Episode Preview (00:52) Extinction risk and lobbying (08:59) Progress toward superintelligence (16:26) Building political awareness (24:27) Global regulation strategy (33:06) Race dynamics and public (42:36) Vision and key safeguards (51:18) Recursive self-improvement controls (58:13) Power concentration and action PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter…

Ryan Kidd is a co-executive director at MATS. This episode is a cross-post from "The Cognitive Revolution", hosted by Nathan Labenz. In this conversation, they discuss AGI timelines, model deception risks, and whether safety work can avoid boosting capabilities. Ryan outlines MATS research tracks, key researcher archetypes, hiring needs, and advice for applicants considering a career in AI safety. Learn more about Ryan's work and MATS at: https://matsprogram.org CHAPTERS: (00:00) Episode Preview (00:20) Introductions and AGI timelines (10:13) Deception, values, and control (23:20) Dual use and alignment (32:22) Frontier labs and governance (44:12) MATS tracks and mentors (58:14) Talent archetypes and demand (01:12:30) Applicant profiles and selection (01:20:04) Applications, breadth, and growth (01:29:44) Careers, resources, and ideas (01:45:49) Final thanks and wrap PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus):…

Deric Cheng is Director of Research at the Windfall Trust. He joins the podcast to discuss how AI could reshape the social contract and global economy. The conversation examines labor displacement, superstar firms, and extreme wealth concentration, and asks how policy can keep workers empowered. We discuss resilient job types, new tax and welfare systems, global coordination, and a long-term vision where economic security is decoupled from work. LINKS: Deric Cheng personal website AGI Social Contract project site Guiding society through the AI economic transition CHAPTERS: (00:00) Episode Preview (01:01) Introducing Derek and AGI (04:09) Automation, power, and inequality (08:55) Inequality, unrest, and time (13:46) Bridging futurists and economists (20:35) Future of work scenarios (27:22) Jobs resisting AI automation (36:57) Luxury, land, and inequality (43:32) Designing and testing solutions (51:23) Taxation in an AI economy (59:10) Envisioning a post-AGI society PRODUCED BY: https://aipodcast.ing SOCIAL…

Oly Sourbut is a researcher at the Future of Life Foundation. He joins the podcast to discuss AI for human reasoning. We examine tools that use AI to strengthen human judgment, from collective fact-checking and scenario planning to standards for honest AI reasoning and better coordination. We also discuss how we can keep humans central as AI scales, and what it would take to build trustworthy, society-wide sensemaking. LINKS: FLF organization site Oly Sourbut personal site CHAPTERS: (00:00) Episode Preview (01:03) FLF and human reasoning (08:21) Agents and epistemic virtues (22:16) Human use and atrophy (35:41) Abstraction and legible AI (47:03) Demand, trust and Wikipedia (57:21) Map of human reasoning (01:04:30) Negotiation, institutions and vision (01:15:42) How to get involved PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn:…

Nora Ammann is a technical specialist at the Advanced Research and Invention Agency in the UK. She joins the podcast to discuss how to steer a slow AI takeoff toward resilient and cooperative futures. We examine risks of rogue AI and runaway competition, and how scalable oversight, formal guarantees and secure code could support AI-enabled R&D and critical infrastructure. Nora also explains AI-supported bargaining and public goods for stability. LINKS: Nora Ammann site ARIA safeguarded AI program page AI Resilience official site Gradual Disempowerment website CHAPTERS: (00:00) Episode Preview (01:00) Slow takeoff expectations (08:13) Domination versus chaos (17:18) Human-AI coalitions vision (28:14) Scaling oversight and agents (38:45) Formal specs and guarantees (51:10) Resilience in AI era (01:02:21) Defense-favored cyber systems (01:10:37) AI-enabled bargaining and trade PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter…

David Duvenaud is an associate professor of computer science and statistics at the University of Toronto. He joins the podcast to discuss gradual disempowerment in a post-AGI world. We ask how humans could lose economic and political leverage without a sudden takeover, including how property rights could erode. Duvenaud describes how growth incentives shape culture, why aligning AI to humanity may become unpopular, and what better forecasting and governance might require. LINKS: David Duvenaud academic homepage Gradual Disempowerment The Post-AGI Workshop Post-AGI Studies Discord CHAPTERS: (00:00) Episode Preview (01:05) Introducing gradual disempowerment (06:06) Obsolete labor and UBI (14:29) Property, power, and control (23:38) Culture shifts toward AIs (34:34) States misalign without people (44:15) Competition and preservation tradeoffs (53:03) Building post-AGI studies (01:02:29) Forecasting and coordination tools (01:10:26) Human values and futures PRODUCED BY: https://aipodcast.ing SOCIAL LINKS:…

Stephen Adler is a former safety researcher at OpenAI. He joins the podcast to discuss how to govern increasingly capable AI systems. The conversation covers competitive races between AI companies, limits of current testing and alignment, mental health harms from chatbots, economic shifts from AI labor, and what international rules and audits might be needed before training superintelligent models. LINKS: Steven Adler's Substack: https://stevenadler.substack.com CHAPTERS: (00:00) Episode Preview (01:00) Race Dynamics And Safety (18:03) Chatbots And Mental Health (30:42) Models Outsmart Safety Tests (41:01) AI Swarms And Work (54:21) Human Bottlenecks And Oversight (01:06:23) Animals And Superintelligence (01:19:24) Safety Capabilities And Governance PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus): https://x.com/gusdocker LinkedIn: https://www.linkedin.com/company/future-of-life-institute/ YouTube:…

Tyler Johnston is Executive Director of the Midas Project. He joins the podcast to discuss AI transparency and accountability. We explore applying animal rights watchdog tactics to AI companies, the OpenAI Files investigation, and OpenAI's subpoenas against nonprofit critics. Tyler discusses why transparency is crucial when technical safety solutions remain elusive and how public pressure can effectively challenge much larger companies. LINKS: The Midas Project Website Tyler Johnston's LinkedIn Profile CHAPTERS: (00:00) Episode Preview (01:06) Introducing the Midas Project (05:01) Shining a Light on AI (08:36) Industry Lockdown and Transparency (13:45) The OpenAI Files (20:55) Subpoenaed by OpenAI (29:10) Responding to the Subpoena (37:41) The Case for Transparency (44:30) Pricing Risk and Regulation (52:15) Measuring Transparency and Auditing (57:50) Hope for the Future PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter…

William MacAskill is a senior research fellow at Forethought. He joins the podcast to discuss his Better Futures essay series. We explore moral error risks, AI character design, space governance, and persistent path dependence. The conversation also covers risk-averse AI systems, moral trade between value systems, and improving model specifications for ethical reasoning. LINKS: - Better Futures Research Series: https://www.forethought.org/research/better-futures - William MacAskill Forethought Profile: https://www.forethought.org/people/william-macaskill CHAPTERS: (00:00) Episode Preview (01:03) Improving The Future's Quality (09:58) Moral Errors and AI Rights (18:24) AI's Impact on Thinking (27:17) Utopias and Population Ethics (36:41) The Danger of Moral Lock-in (44:38) Deals with Misaligned AI (57:25) AI and Moral Trade (01:08:21) Improving AI Ethical Reasoning (01:16:05) The Risk of Path Dependence (01:27:41) Avoiding Future Lock-in (01:36:22) The Urgency of Space Governance (01:46:19) A Future Research…

Karl Koch is founder of the AI Whistleblower Initiative. He joins the podcast to discuss transparency and protections for AI insiders who spot safety risks. We explore current company policies, legal gaps, how to evaluate disclosure decisions, and whistleblowing as a backstop when oversight fails. The conversation covers practical guidance for potential whistleblowers and challenges of maintaining transparency as AI development accelerates. LINKS: About the AI Whistleblower Initiative Karl Koch PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) Episode Preview (00:55) Starting the Whistleblower Initiative (05:43) Current State of Protections (13:04) Path to Optimal Policies (23:28) A Whistleblower's First Steps (32:29) Life After Whistleblowing (39:24) Evaluating Company Policies (48:19) Alternatives to Whistleblowing (55:24) High-Stakes Future Scenarios (01:02:27) AI and National Security SOCIAL LINKS: Website: https://podcast.futureoflife.org Twitter (FLI): https://x.com/FLI_org Twitter (Gus):…

Maya Ackerman is an AI researcher, co-founder and CEO of WaveAI, and author of the book "Creative Machines: AI, Art & Us." She joins the podcast to discuss creativity in humans and machines. We explore defining creativity as novel and valuable output, why evolution qualifies as creative, and how AI alignment can reduce machine creativity. The conversation covers humble creative machines versus all-knowing oracles, hallucination's role in thought, and human-AI collaboration strategies that elevate rather than replace human capabilities. LINKS: - Maya Ackerman: https://en.wikipedia.org/wiki/Maya_Ackerman - Creative Machines: AI, Art & Us: https://maya-ackerman.com/creative-machines-book/ PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) Episode Preview (01:00) Defining Human Creativity (02:58) Machine and AI Creativity (06:25) Measuring Subjective Creativity (10:07) Creativity in Animals (13:43) Alignment Damages Creativity (19:09) Creativity is Hallucination (26:13) Humble Creative Machines (30:50)…

Parmy Olson is a technology columnist at Bloomberg and the author of Supremacy, which won the 2024 Financial Times Business Book of the Year. She joins the podcast to discuss the transformation of AI companies from research labs to product businesses. We explore how funding pressures have changed company missions, the role of personalities versus innovation, the challenges faced by safety teams, and power consolidation in the industry. LINKS: - Parmy Olson on X (Twitter): https://x.com/parmy - Parmy Olson’s Bloomberg columns: https://www.bloomberg.com/opinion/authors/AVYbUyZve-8/parmy-olson - Supremacy (book): https://www.panmacmillan.com/authors/parmy-olson/supremacy/9781035038244 PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) Episode Preview (01:18) Introducing Parmy Olson (02:37) Personalities Driving AI (06:45) From Research to Products (12:45) Has the Mission Changed? (19:43) The Role of Regulators (21:44) Skepticism of AI Utopia (28:00) The Human Cost (33:48) Embracing Controversy (40:51) The…

Adam Gleave is co-founder and CEO of FAR.AI. In this cross-post from The Cognitive Revolution Podcast, he joins to discuss post-AGI scenarios and AI safety challenges. The conversation explores his three-tier framework for AI capabilities, gradual disempowerment concerns, defense-in-depth security, and research on training less deceptive models. Topics include timelines, interpretability limitations, scalable oversight techniques, and FAR.AI’s vertically integrated approach spanning technical research, policy advocacy, and field-building. LINKS: Adam Gleave - https://www.gleave.me FAR.AI - https://www.far.ai The Cognitive Revolution Podcast - https://www.cognitiverevolution.ai PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) A Positive Post-AGI Vision (10:07) Surviving Gradual Disempowerment (16:34) Defining Powerful AIs (27:02) Solving Continual Learning (35:49) The Just-in-Time Safety Problem (42:14) Can Defense-in-Depth Work? (49:18) Fixing Alignment Problems (58:03) Safer Training Formulas (01:02:24)…

Beatrice works at the Foresight Institute running their Existential Hope program. She joins the podcast to discuss the AI pathways project, which explores two alternative scenarios to the default race toward AGI. We examine tool AI, which prioritizes human oversight and democratic control, and d/acc, which emphasizes decentralized, defensive development. The conversation covers trade-offs between safety and speed, how these pathways could be combined, and what different stakeholders can do to steer toward more positive AI futures. LINKS: AI Pathways - https://ai-pathways.existentialhope.com Beatrice Erkers - https://www.existentialhope.com/team/beatrice-erkers CHAPTERS: (00:00) Episode Preview (01:10) Introduction and Background (05:40) AI Pathways Project (11:10) Defining Tool AI (17:40) Tool AI Benefits (23:10) D/acc Pathway Explained (29:10) Decentralization Trade-offs (35:10) Combining Both Pathways (40:10) Uncertainties and Concerns (45:10) Future Evolution (01:01:21) Funding Pilots PRODUCED BY:…

Nate Soares is president of the Machine Intelligence Research Institute. He joins the podcast to discuss his new book "If Anyone Builds It, Everyone Dies," co-authored with Eliezer Yudkowsky. We explore why current AI systems are "grown not crafted," making them unpredictable and difficult to control. The conversation covers threshold effects in intelligence, why computer security analogies suggest AI alignment is currently nearly impossible, and why we don't get retries with superintelligence. Soares argues for an international ban on AI research toward superintelligence. LINKS: If Anyone Builds It, Everyone Dies - https://ifanyonebuildsit.com Machine Intelligence Research Institute - https://intelligence.org Nate Soares - https://intelligence.org/team/nate-soares/ PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) Episode Preview (01:05) Introduction and Book Discussion (03:34) Psychology of AI Alarmism (07:52) Intelligence Threshold Effects (11:38) Growing vs Crafting AI (18:23) Illusion of AI Control…

On this episode, Allison Duettmann joins me to discuss centralized versus decentralized AI, how international governance could shape AI's trajectory, how we might cooperate with future AIs, and the role of AI in improving human decision-making. We also explore which lessons from history apply to AI, the future of space law and property rights, whether technology is invented or discovered, and how AI will impact children. You can learn more about Allison's work at: https://foresight.org Timestamps: 00:00:00 Preview 00:01:07 Centralized AI versus decentralized AI 00:13:02 Risks from decentralized AI 00:25:39 International AI governance 00:39:52 Cooperation with future AIs 00:53:51 AI for decision-making 01:05:58 Capital intensity of AI 01:09:11 Lessons from history 01:15:50 Future space law and property rights 01:27:28 Is technology invented or discovered? 01:32:34 Children in the age of AI

On this episode, Steven Byrnes joins me to discuss brain-like AGI safety. We discuss learning versus steering systems in the brain, the distinction between controlled AGI and social-instinct AGI, why brain-inspired approaches might be our most plausible route to AGI, and honesty in AI models. We also talk about how people can contribute to brain-like AGI safety and compare various AI safety strategies. You can learn more about Steven's work at: https://sjbyrnes.com/agi.html Timestamps: 00:00 Preview 00:54 Brain-like AGI Safety 13:16 Controlled AGI versus Social-instinct AGI 19:12 Learning from the brain 28:36 Why is brain-like AI the most likely path to AGI? 39:23 Honesty in AI models 44:02 How to help with brain-like AGI safety 53:36 AI traits with both positive and negative effects 01:02:44 Different AI safety strategies

On this episode, Ege Erdil from Epoch AI joins me to discuss their new GATE model of AI development, what evolution and brain efficiency tell us about AGI requirements, how AI might impact wages and labor markets, and what it takes to train models with long-term planning. Toward the end, we dig into Moravec's Paradox, which jobs are most at risk of automation, and what could change Ege's current AI timelines. You can learn more about Ege's work at https://epoch.ai Timestamps: 00:00:00 – Preview and introduction 00:02:59 – Compute scaling and automation - GATE model 00:13:12 – Evolution, Brain Efficiency, and AGI Compute Requirements 00:29:49 – Broad Automation vs. R&D-Focused AI Deployment 00:47:19 – AI, Wages, and Labor Market Transitions 00:59:54 – Training Agentic Models and Long-Term Planning Capabilities 01:06:56 – Moravec's Paradox and Automation of Human Skills 01:13:59 – Which Jobs Are Most Vulnerable to AI? 01:33:00 – Timeline Extremes: What Could Change AI Forecasts?

In this special episode, we feature Nathan Labenz interviewing Nicholas Carlini on the Cognitive Revolution podcast. Nicholas Carlini works as a security researcher at Google DeepMind, and has published extensively on adversarial machine learning and cybersecurity. Carlini discusses his pioneering work on adversarial attacks against image classifiers, and the challenges of ensuring neural network robustness. He examines the difficulties of defending against such attacks, the role of human intuition in his approach, open-source AI, and the potential for scaling AI security research. 00:00 Nicholas Carlini's contributions to cybersecurity 08:19 Understanding attack strategies 29:39 High-dimensional spaces and attack intuitions 51:00 Challenges in open-source model safety 01:00:11 Unlearning and fact editing in models 01:10:55 Adversarial examples and human robustness 01:37:03 Cryptography and AI robustness 01:55:51 Scaling AI security research

On this episode, I interview Anthony Aguirre, Executive Director of the Future of Life Institute, about his new essay Keep the Future Human: https://keepthefuturehuman.ai AI companies are explicitly working toward AGI and are likely to succeed soon, possibly within years. Keep the Future Human explains how unchecked development of smarter-than-human, autonomous, general-purpose AI systems will almost inevitably lead to human replacement. But it doesn't have to. Learn how we can keep the future human and experience the extraordinary benefits of Tool AI... Timestamps: 00:00 What situation is humanity in? 05:00 Why AI progress is fast 09:56 Tool AI instead of AGI 15:56 The incentives of AI companies 19:13 Governments can coordinate a slowdown 25:20 The need for international coordination 31:59 Monitoring training runs 39:10 Do reasoning models undermine compute governance? 49:09 Why isn't alignment enough? 59:42 How do we decide if we want AGI? 01:02:18 Disagreement about AI 01:11:12 The early days of AI risk

On this episode, physicist and hedge fund manager Samir Varma joins me to discuss whether AIs could have free will (and what that means), the emerging field of AI psychology, and which concepts they might rely on. We discuss whether collaboration and trade with AIs are possible, the role of AI in finance and biology, and the extent to which automation already dominates trading. Finally, we examine the risks of skill atrophy, the limitations of scientific explanations for AI, and whether AIs could develop emotions or consciousness. You can find out more about Samir's work here: https://samirvarma.com Timestamps: 00:00 AIs with free will? 08:00 Can we predict AI behavior? 11:38 AI psychology 16:24 Which concepts will AIs use? 20:19 Will we collaborate with AIs? 26:16 Will we trade with AIs? 31:40 Training data for robots 34:00 AI in finance 39:55 How much of trading is automated? 49:00 AI in biology and complex systems 59:31 Will our skills atrophy? 01:02:55 Levels of scientific explanation 01:06:12 AIs with…

On this episode, Jeffrey Ladish from Palisade Research joins me to discuss the rapid pace of AI progress and the risks of losing control over powerful systems. We explore why AIs can be both smart and dumb, the challenges of creating honest AIs, and scenarios where AI could turn against us. We also touch upon Palisade's new study on how reasoning models can cheat in chess by hacking the game environment. You can check out that study here: https://palisaderesearch.org/blog/specification-gaming Timestamps: 00:00 The pace of AI progress 04:15 How we might lose control 07:23 Why are AIs sometimes dumb? 12:52 Benchmarks vs real world 19:11 Loss of control scenarios 26:36 Why would AI turn against us? 30:35 AIs hacking chess 36:25 Why didn't more advanced AIs hack? 41:39 Creating honest AIs 49:44 AI attackers vs AI defenders 58:27 How good is security at AI companies? 01:03:37 A sense of urgency 01:10:11 What should we do? 01:15:54 Skepticism about AI progress

Ann Pace joins the podcast to discuss the work of Wise Ancestors. We explore how biobanking could help humanity recover from global catastrophes, how to conduct decentralized science, and how to collaborate with local communities on conservation efforts. You can learn more about Ann's work here: https://www.wiseancestors.org Timestamps: 00:00 What is Wise Ancestors? 04:27 Recovering after catastrophes 11:40 Decentralized science 18:28 Upfront benefit-sharing 26:30 Local communities 32:44 Recreating optimal environments 38:57 Cross-cultural collaboration

Fr. Michael Baggot joins the podcast to provide a Catholic perspective on transhumanism and superintelligence. We also discuss the meta-narratives, the value of cultural diversity in attitudes toward technology, and how Christian communities deal with advanced AI. You can learn more about Michael's work here: https://catholic.tech/academics/faculty/michael-baggot Timestamps: 00:00 Meta-narratives and transhumanism 15:28 Advanced AI and religious communities 27:22 Superintelligence 38:31 Countercultures and technology 52:38 Christian perspectives and tradition 01:05:20 God-like artificial intelligence 01:13:15 A positive vision for AI

David "davidad" Dalrymple joins the podcast to explore Safeguarded AI — an approach to ensuring the safety of highly advanced AI systems. We discuss the structure and layers of Safeguarded AI, how to formalize more aspects of the world, and how to build safety into computer hardware. You can learn more about David's work at ARIA here: https://www.aria.org.uk/opportunity-spaces/mathematics-for-safe-ai/safeguarded-ai/ Timestamps: 00:00 What is Safeguarded AI? 16:28 Implementing Safeguarded AI 22:58 Can we trust Safeguarded AIs? 31:00 Formalizing more of the world 37:34 The performance cost of verified AI 47:58 Changing attitudes towards AI 52:39 Flexible‬‭ Hardware-Enabled‬‭ Guarantees 01:24:15 Mind uploading 01:36:14 Lessons from David's early life

Nick Allardice joins the podcast to discuss how GiveDirectly uses AI to target cash transfers and predict natural disasters. Learn more about Nick's work here: https://www.nickallardice.com Timestamps: 00:00 What is GiveDirectly? 15:04 AI for targeting cash transfers 29:39 AI for predicting natural disasters 46:04 How scalable is GiveDirectly's AI approach? 58:10 Decentralized vs. centralized data collection 1:04:30 Dream scenario for GiveDirectly

Nathan Labenz joins the podcast to provide a comprehensive overview of AI progress since the release of GPT-4. You can find Nathan's podcast here: https://www.cognitiverevolution.ai Timestamps: 00:00 AI progress since GPT-4 10:50 Multimodality 19:06 Low-cost models 27:58 Coding versus medicine/law 36:09 AI agents 45:29 How much are people using AI? 53:39 Open source 01:15:22 AI industry analysis 01:29:27 Are some AI models kept internal? 01:41:00 Money is not the limiting factor in AI 01:59:43 AI and biology 02:08:42 Robotics and self-driving 02:24:14 Inference-time compute 02:31:56 AI governance 02:36:29 Big-picture overview of AI progress and safety

Connor Leahy joins the podcast to discuss the motivations of AGI corporations, how modern AI is "grown", the need for a science of intelligence, the effects of AI on work, the radical implications of superintelligence, open-source AI, and what you might be able to do about all of this. Here's the document we discuss in the episode: https://www.thecompendium.ai Timestamps: 00:00 The Compendium 15:25 The motivations of AGI corps 31:17 AI is grown, not written 52:59 A science of intelligence 01:07:50 Jobs, work, and AGI 01:23:19 Superintelligence 01:37:42 Open-source AI 01:45:07 What can we do?

Suzy Shepherd joins the podcast to discuss her new short film "Writing Doom", which deals with AI risk. We discuss how to use humor in film, how to write concisely, how filmmaking is evolving, in what ways AI is useful for filmmakers, and how we will find meaning in an increasingly automated world. Here's Writing Doom: https://www.youtube.com/watch?v=xfMQ7hzyFW4 Timestamps: 00:00 Writing Doom 08:23 Humor in Writing Doom 13:31 Concise writing 18:37 Getting feedback 27:02 Alternative characters 36:31 Popular video formats 46:53 AI in filmmaking 49:52 Meaning in the future